Comments (4)
SharedHashMap had a bug in it's iterator, perhaps HugehashMap does as well.
We will check this.
On 20 August 2014 19:54, Mignastor [email protected] wrote:
Hi everyone,
i am having an issue with my HugeHashMap and Database persist.
In fact, when i persist my objects from the HugeHashMap i don't get what i
was expecting.I have an Instance that contains an id, a word context as a String, some
positions as integers,..
And instead of having the values i got this for each Instance in
HugeHashMap:
id="table.Instance[ id=000171fe-15a2-46fd-97eb-957d42940f8a ]"
Auto="0"
Context=""
position1="0"
position2="0"
position3="0"
position4="0"
position5="0"
foreignId=""Here is the code where i manage the HugeHashMap:
public void generateInstances(String path) throws IOException, ParseException, InvalidTokenOffsetsException{
HugeConfig config = HugeConfig.DEFAULT.clone();
HugeHashMap<String, Instance> instanceArray = new HugeHashMap<String, Instance>(config,String.class,Instance.class);
HugeHashMap<Integer,TermeinstancedocumentPK> tidArray = new HugeHashMap<Integer,TermeinstancedocumentPK>(config,Integer.class,TermeinstancedocumentPK.class);
ArrayList contexesArray = new ArrayList<>();
Integer idtid=1;
DocumentIndexer lucene = new DocumentIndexer(path);
Project p = this.getProject();
for(Notion n : p.getNotionCollection()){
for (Term t : n.getTermCollection()) {
for (Variable v : t.getVariableCollection()) {
contexesArray = lucene.getContextes(v.getName(),90);
for(Context c : contexesArray){
boolean instanceAlreadyExists=false;
String rightContext = c.getRightContext();
String[] contextRight = rightContext.split("\n");
StringJoiner joiner = new StringJoiner(" | ");
for(String s: contextRight){
joiner.add(s);
}
rightContext = joiner.toString();
String leftContext = c.getLeftContext();
int leftContextSizeBefore=leftContext.length();
String[] contextLeft = leftContext.split("\n");
StringJoiner joiner2 = new StringJoiner(" | ");
for(String s: contextLeft){
joiner2.add(s);
}
leftContext = joiner2.toString();
int leftContextSizeAfter=leftContext.length();
int relativeLeftContextSize = leftContextSizeAfter-leftContextSizeBefore;
String context = leftContext + c.getTerm() + rightContext;
Instance i = new Instance();
if(instanceArray.containsKey(context)){
instanceAlreadyExists=true;
i=instanceArray.get(context);
}
if(!instanceAlreadyExists){
String idinst = UUID.randomUUID().toString();
i.setId(idinst);
i.setContexte(context);
i.setPositionBeginTermeInstance(c.getStartTerm()+relativeLeftContextSize);
i.setPositionEndTermeInstance(c.getEndTerm()+relativeLeftContextSize);
i.setPositionBeginTermeTxtBrut(c.getStartTerm());
i.setPositionEndTermeTxtBrut(c.getEndTerm());
i.setContexteSize(c.getContext().length());
i.setProjectId(p);
if((p.getNbWordMinForAuto()!=-1 &&(this.getNbWord(v.getName())>p.getNbWordMinForAuto()))
| (p.getNbCharMinForAuto()!=-1 && (this.getNbChar(v.getName())>p.getNbCharMinForAuto()))){
i.setAuto(true);
}
TermeinstancedocumentPK tidPK = new TermeinstancedocumentPK(t.getId(),c.getUuid(), idinst,c.getIndexStart());
instanceArray.put(context,i);
tidArray.put(idtid,tidPK);
idtid++;
}
else{
if((p.getNbWordMinForAuto()!=-1 &&(this.getNbWord(v.getName())>p.getNbWordMinForAuto()))
| (p.getNbCharMinForAuto()!=-1 && (this.getNbChar(v.getName())>p.getNbCharMinForAuto()))){
instanceArray.get(context).setAuto(true);
}
TermeinstancedocumentPK tidPK = new TermeinstancedocumentPK(t.getId(),c.getUuid(), i.getId(),c.getIndexStart());
tidArray.put(idtid,tidPK);
idtid++;
}
}
contexesArray.clear();
}
}
}
lucene.close();
this.addInstances(instanceArray);
this.addTIDs(tidArray);
instanceArray.clear();
tidArray.clear();
}
//the method to persist is like thatpublic void addInstances(HugeHashMap<String,Instance> ints){
this.open();
em.getTransaction().begin();
for(Instance inst : ints.values()){
em.persist(inst);
}
em.getTransaction().commit();
this.close();
}Do you have any idea to solve this problem?
Thanks
—
Reply to this email directly or view it on GitHub
#41.
from hugecollections-old.
Hi,
Thanks for your quick answer.
I made some tests and it appears that i can't iterate throught the HugeHashMap.
To solve this problem i had to put an Integer as first parameter and do like this:
HugeHashMap<Integer, HashMap<String,String>>...
int size=ints.size();
for(int i=1;i<size;i++){
HashMap<String,String> instanceMap=ints.get(i);
...
Also i had to replace JPA items by an HashMap<String,String> with each Key, value i need to create the JPA Object. Then, i create all objects when i iterate the HugeHashMap and persist them.
In fact, JPA Objects are not restored correctly from HugeHashMap by the get function and that is why i had wrong values.
Do you have any idea why or is it a problem with JPA?
Thanks
from hugecollections-old.
Thanks for your feedback regarding the iterator, we will take a look to see if we can fix this issue, just as an aside we are no longer doing any new development on HugeHashHap, we have moved our development focus to ChronicleMap, which ( apart from resizing capability should offer you most of the functionality that you get with HugeHashMap ), it should also not have the iterator issue that you identified above.
Regarding your JPA issue, HugeHashMap stores the keys and values off heap, so the JPA object will undergo serialization, The problems with JPA maybe down to serialization.
from hugecollections-old.
we have raised the following JIRA to look at the iterator issue see https://higherfrequencytrading.atlassian.net/browse/HCOLL-126
from hugecollections-old.
Related Issues (20)
- setSegments(int) is not public in HugeConfig; cannot be accessed from outside package HOT 3
- Comparing behaviour of HugeHashMap with HashMap HOT 2
- custom marshalling/serialization HOT 16
- remark on replication/distribution HOT 9
- how to get exactly 1 OpenHFT test to run HOT 10
- Using Shared/HugeHashMap HOT 3
- PingPong latency test across /dev/shm/SharedHashMap IPC ... HOT 19
- support for "read through" and "write behind" gettr/settrs (in addition to JBI) HOT 25
- Cache size exceed bug HOT 1
- TCPSocketReplication4WayMapTest testBufferOverflow failed (Windows 8.1) HOT 10
- VanillaShortShortMultiMap is full when capacity is still available HOT 5
- Difference between setSmallEntrySize and setCapacity
- SharedHashMap: iterating using keyset, valueset is very slow for larger tables HOT 8
- problem using SharedHashMapBuilder ... HOT 6
- OpenHFT scope of capability re: SHMEntry fields (user JBI supplied) and "barrier" operations HOT 9
- SharedHashMap causing crash on Windows OS HOT 28
- Chronicle Map not thread-safe in certain circumstances HOT 14
- ChronicleMap / ChronicleMapBuilder - actualEntriesPerSegment restricted to minimum of 8 HOT 13
- too many Segment.getKey() invocations HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from hugecollections-old.