Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded about hdt-java HOT 8 CLOSED

You can crank up the heap space (default: 1GB), eg. to 20GB, by prepending your command line using JAVA_OPTIONS=-Xmx20g.

from hdt-java.

Make sure your configuration uses the "Big" dictionary (https://github.com/rdfhdt/hdt-java/blob/master/hdt-java-core/src/main/java/org/rdfhdt/hdt/dictionary/impl/FourSectionDictionaryBig.java) , should be enabled by default though. (https://github.com/rdfhdt/hdt-java/blob/master/hdt-java-core/hdt.cfg)

I would recommend using the CPP version though, it's more efficient, faster and more stable.

from hdt-java.

I would recommend using the CPP version though, it's more efficient, faster and more stable.

I got the impression that the CPP version was outdated? You said "You probably need to generate the HDT file with java too; both libraries might no longer be compatible."

Plus, looks like the CPP version doesn't compile?

from hdt-java.

The CPP version definitely has more development going on at the moment, that's why it probably diverted from the java version. I haven't checked, so the HDT's could still be compatible. It does compile for most people AFAIK.

Back to your issue, I honestly don't know. I wasn't involved in the development. But I can imagine that large datasets are a problem for the current implementation, that's why https://github.com/rdfhdt/hdt-mr came about. I could also be an issue with your Garbage Collection configuration.

from hdt-java.

I haven't checked, so the HDT's could still be compatible

regardless if a .hdt was created with C++ or Java, shouldn't it be the exact same format?

from hdt-java.

in theory yes, in practice not that easy without dedicated development teams :/

from hdt-java.

Sorry I didn't want to imply that you are not doing enough, I just thought that HDT was more or less a well defined standard and for this reason shouldn't change.

from hdt-java.

No worries, you didn't imply anything. I just wanted to explain we want the same thing, but are dependent on charity :)

from hdt-java.