Comments (3)
We support off heap references. You can do this yourself by implementing
the Byteable interface or you can use our generated classes by defining an
interface of getter/setters/adders etc with nested classes and arrays.
SharedHashMap supports concurrent access across processes not just threads
without the need for copying.
You can add entry based locking or use CAS or atomic add operations.
If you over commit the size, add lots of entries and then delete most of
them, it wont compact the entries left. Ie they never move so you can hold
references to them.
Without tuning the system you cannot have more than 32k files mapped at
once. I suggest you have less than a hundred to keep things simple in terms
of management etc.
On 30/05/2014 9:29 PM, "flxthomaslo" [email protected] wrote:
looking into using shared or huge hash map within our application and got
couple of questions:
We have our binary encoded data and we use flyweight to access it. If
we are using huge hash map assuming we only access the data via a single
thread, what is the best way to access the binary data without copying it
(basically we only need the starting address of the value then we can map
the flyweight to it)Similar question on using shared map on using flyweight to read the
information. I assume since we will be using multiple thread reading it we
will need to do copy inFor the shared map is there a true locking (all read and write will
block each other?)If used for a long time is there a memory fragmentation we need to
worry about?Is it ok to have multiple shared map backed by multiple memory map
files in terms of OS stability?Thanks very much for such a great work!
—
Reply to this email directly or view it on GitHub
#27.
from hugecollections-old.
I probably won't use the generated classes since we have our own. Will take a look at byteable. For locking sounds like we can customize it (not sure how yet but will take a look). If we are only using it for read only I don't think I want to put a true lock in there to prevent other processes to read since for our use case it is acceptable to be eventually consistent (don't need to be atomic for such access). But then I will need to perform copy. So will need to see the trade off (read access time versus copy time). What type of tuning would you recommend? I think hundred should do.
from hugecollections-old.
Volatile and lazy set is also supported. You can use timestamp locking for
optimistic locking. The cost of locking and unlocking a record is about 50
ns. If you use a timstamp lock it is more complex but with high read
access the cost drops to about 10 ns.
On 31/05/2014 4:02 PM, "flxthomaslo" [email protected] wrote:
I probably won't use the generated classes since we have our own. Will
take a look at byteable. For locking sounds like we can customize it (not
sure how yet but will take a look). If we are only using it for read only I
don't think I want to put a true lock in there to prevent other processes
to read since for our use case it is acceptable to be eventually consistent
(don't need to be atomic for such access). But then I will need to perform
copy. So will need to see the trade off (read access time versus copy
time). What type of tuning would you recommend? I think hundred should do.—
Reply to this email directly or view it on GitHub
#27 (comment)
.
from hugecollections-old.
Related Issues (20)
- setSegments(int) is not public in HugeConfig; cannot be accessed from outside package HOT 3
- Comparing behaviour of HugeHashMap with HashMap HOT 2
- custom marshalling/serialization HOT 16
- remark on replication/distribution HOT 9
- how to get exactly 1 OpenHFT test to run HOT 10
- PingPong latency test across /dev/shm/SharedHashMap IPC ... HOT 19
- support for "read through" and "write behind" gettr/settrs (in addition to JBI) HOT 25
- Cache size exceed bug HOT 1
- TCPSocketReplication4WayMapTest testBufferOverflow failed (Windows 8.1) HOT 10
- VanillaShortShortMultiMap is full when capacity is still available HOT 5
- Difference between setSmallEntrySize and setCapacity
- SharedHashMap: iterating using keyset, valueset is very slow for larger tables HOT 8
- problem using SharedHashMapBuilder ... HOT 6
- OpenHFT scope of capability re: SHMEntry fields (user JBI supplied) and "barrier" operations HOT 9
- HugeHashMap with JPA elements HOT 4
- SharedHashMap causing crash on Windows OS HOT 28
- Chronicle Map not thread-safe in certain circumstances HOT 14
- ChronicleMap / ChronicleMapBuilder - actualEntriesPerSegment restricted to minimum of 8 HOT 13
- too many Segment.getKey() invocations HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from hugecollections-old.