Comments (9)
We have been working on a similar solution over the last few weeks.
We have some test cases for TCP Replication, see for example :
see net.openhft.collections.replication.TCPSocketReplicationTest3way
We have also started work on UDP replication,
Although we have not yet completed testing this code : net.openhft.collections.replication.UDPSocketReplicationTest#testBufferOverflow
We have just check this code into:
<groupId>net.openhft</groupId>
<artifactId>collections</artifactId>
<version>3.1.1-SNAPSHOT</version>
So it should be available for download as a snapshot, in the next few hours.
Rob
On 27 May 2014, at 17:46, RuedigerMoeller [email protected] wrote:
If you manage to define kind of in/out interfaces for replication messages, I could plug in a multicast based replication, which is favourable over TCP in many setups.
—
Reply to this email directly or view it on GitHub.
from hugecollections-old.
Hi all,
I think the "distribution" part of Ruediger's title "remark on distribution/replication" deserves some content too!
Any comment wrt to OpenHFT ambitions to provide an explicit DistributedSharedHashMap interface? Where in fact the "Map<K,V>" set could be partitioned (instead of replicated) and synchronized across host separated compute nodes ... empowering PB+ capacity Maps?
Just now getting back into my "what's new at OpenHFT?" mode, I see a whole lot.
- SHM Listeners
- SHM Replicators
- TCP accommodations
- UDP accommodations
- IPC accommodations
Being blunt, if you then add DistributedSHM as operand view, OpenHFT is just a few details (on the operator side) from itself being a (highly preferred) JCACHE/JGRID provider.
from hugecollections-old.
@Cotton-Ben isn't distribution is Hazelcast's field?
from hugecollections-old.
Hazelcast does provided a distributed view of my K,V data set ... but ... they use only {UDP,TCP} as a transport set ... and for our supercomputer grid (distributed to only locality=127.0.0.1, no NIC hops) the use of only {UDP,TCP} as transport is a delinquency (we ain't going to dance up and down the OSI stack if there is no NIC hop). OpenHFT solves this because we can use /dev/shm as an IPC transport. So, I'm still saying, given that y'all already solve our locality=localhost transport needs ... we'd like to audition you as a potential distributed operand provider ... empowering us to kill 2 birds with 1 OpenHFT stone.
from hugecollections-old.
@rob Just my 2 pence:
A reliable UDP low latency messaging is quite some effort (including retransmission on packet loss, message recording, total ordering for transparent failover, high avaiability ..).
Additionally: large systems (and these are the ones requiring multicast-udp and off-heap storage) most of the time already have a udp-message-bus system. E.g. my distributed datagrid cluster @ Eurex runs on a commercial low latency udp messaging solution in production, fast-cast 1.0 (slow, but small footprint) in development/functional test, fast-cast 2.0 is considered for other, smaller exchange systems to avoid license fees.
So imo best thing would be to have kind of a public transport interface defined. Ofc you still can provide your own implementations.
@Cotton-Ben I have built a distributed, continuous query datagrid running in production @ Eurex Exchange featuring evenly sharded data across all nodes.
From my POV, OpenHFT shouldn't mud up layers. Providing IPC Off-Heap storage is one thing. Implementing replication another thing. Sharding data amongst distributed instances is then on top.
I currently evaluate to use SharedMemHashMap as a base storage foundation for offheap data (on heap data is still used for low latency market data distribution). So in my use case, messaging, sharding, streaming, querying, high avaiability is already in place.
In general an open source, high efficient distributed continuous datagrid ofc would be a great product as the performance of redis/mongo & co seems not-that good compared to the what is possible. Also they don't feature continuous queries, which is a must-have feature to scale large realtime systems.
Funny enough I am currently also trending towards something like this with my os stuff :-) [will take years, I am currently stuck serializing java<=>JS<=>Dart its really ugly, but JSon does not cut it]
The closed source stuff I did the last years turned out to be amazing (not because of super efficient implementation, but its very easy to extend and build stuff on top). Continuous distributed datagrids can act as a large-scale model to back GUI clients, backoffice systems and realtime computation. Its MVC at large scale.
from hugecollections-old.
@leventov only if you are willing to replace 'Million' by 1000.
from hugecollections-old.
@RuedigerMoeller interesting ... nice job with EurexExchange delivery. I see your point it is a lot of work. For me, I face a race-condition ... from which whoever wins I will be happy.
Racer 1= some JCACHE/JGRID vendor adopting OpenHFT as their Off-Heap ZERO-COPY /dev/shm Java IPC transport provider
Racer 2=OpenHFT passing the 107/347 TCKs and itself becoming a JCACHE/JGRID provider
from hugecollections-old.
@RuedigerMoeller ( also posted here as we appear to have 2 threads going on )
you may find this interface useful
net.openhft.collections.ReplicatedSharedHashMap.EntryExternalizable
its implemented by :
net.openhft.collections.VanillaSharedReplicatedHashMap
from hugecollections-old.
@Cotton-Ben seems i need some googling ;)
@BoundedBuffer Thanks for the hint, I'll cose this one.
from hugecollections-old.
Related Issues (20)
- setSegments(int) is not public in HugeConfig; cannot be accessed from outside package HOT 3
- Comparing behaviour of HugeHashMap with HashMap HOT 2
- custom marshalling/serialization HOT 16
- how to get exactly 1 OpenHFT test to run HOT 10
- Using Shared/HugeHashMap HOT 3
- PingPong latency test across /dev/shm/SharedHashMap IPC ... HOT 19
- support for "read through" and "write behind" gettr/settrs (in addition to JBI) HOT 25
- Cache size exceed bug HOT 1
- TCPSocketReplication4WayMapTest testBufferOverflow failed (Windows 8.1) HOT 10
- VanillaShortShortMultiMap is full when capacity is still available HOT 5
- Difference between setSmallEntrySize and setCapacity
- SharedHashMap: iterating using keyset, valueset is very slow for larger tables HOT 8
- problem using SharedHashMapBuilder ... HOT 6
- OpenHFT scope of capability re: SHMEntry fields (user JBI supplied) and "barrier" operations HOT 9
- HugeHashMap with JPA elements HOT 4
- SharedHashMap causing crash on Windows OS HOT 28
- Chronicle Map not thread-safe in certain circumstances HOT 14
- ChronicleMap / ChronicleMapBuilder - actualEntriesPerSegment restricted to minimum of 8 HOT 13
- too many Segment.getKey() invocations HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from hugecollections-old.