afs / mantis Goto Github PK

TDB2 has a reorganized NodeId which usually has space for 56 bit encoded values inline. It also has space for 62 bits for the one case of xsd:double but there is no implementation yet.

• Details of xsd:double, a IEEE 754 binary64.
• Wikipedia entry about IEEE 754-2008 binary64.

To inline the value without increasing the size of NodeId, 2 bits need to be saved. Values not meeting the restriction will be stored as nodes in the node table.

Proposal: keep the significand (mantissa) as 52 bits and reduce the exponent from 11 bits to 9 bits - making a size limit of inlined values 1.157921e+77, rather than the full 1.797693e+308.

Infinities, +/- zero and NaNs are inlined.

This is recoverable - inline values will be perfected recovered.

Reducing the precision instead risks looses accuracy.

TDB2 already added the inlining of xsd:floats and retaining the datatypes of integer derived datatypes.

Initial testing of TDB2 loading 9M triples fails after 1.4M with disk full - 16GB. This is with jena-fuseki 3.5.0-SNAPSHOT + TDB2 as described below.

Configuration with TDB and released Jena 3.4.0 loads to completion with total of 3.1GB in use.

I observe that TDB2 is allocating disk space very rapidly during loading. The loading is being done via the graph store protocol, org.apache.jena.query.DatasetAccessor.putModel for each of 146K named graphs.

Am I missing a configuration piece or is this perhaps an interaction with something in Jena 3.5.0-SNAPSHOT?

I cloned mantis and did

mvn install 

and then cloned jena (3.5.0-SNAPSHOT), updated the jena-fuseki-war pom.xml with the dependency and did a

mvn install 

in jena and

mvn package 

in jena-fuseki-war and produced the desired war file with tdb2-0.3.0.jar.

I then updated the configuration file with:

@prefix tdb2:    <http://jena.apache.org/2016/tdb#> .

. . .

[] ja:loadClass "org.seaborne.tdb2.TDB2" .
tdb2:DatasetTDB2  rdfs:subClassOf  ja:RDFDataset .

. . .

:dataset_test rdf:type      tdb2:DatasetTDB2 ;
     tdb2:location "/etc/fuseki/databases/test" ;
     tdb2:unionDefaultGraph true ;

Java is 1.8.0_144 on Debian 4.9.30-2+deb9u3 (2017-08-06) x86_64 GNU/Linux and Tomcat 8.0.12.

Here's the complete fuseki config: bdrc.ttl.zip