Processing of siva files bigger than 2Gb about gemini HOT 7 CLOSED

Good point haha. I honestly believed I did, but no. Here you go: src-d/siva-java#18

from gemini.

Siva reader uses MappedByteBuffer:
https://github.com/src-d/siva-java/blob/master/src/main/java/tech/sourced/siva/SivaReader.java#L43

And there is a limit in jdk https://github.com/openjdk-mirror/jdk7u-jdk/blob/master/src/share/classes/sun/nio/ch/FileChannelImpl.java#L788

Then there is an issue in Spark, it can't have partition more than 2G:
https://issues.apache.org/jira/browse/SPARK-6235
The issue is open since 2015 year but no much progress since then.

We can't change it.

@bzz @dpordomingo

from gemini.

2 issues here:

1. siva-java fails to read .siva files > 2Gb

   To understand, if that is by-design, or is it a 🐛 I would suggest

   • find a repository that results in > 2Gb .siva file
   • borges pack it locally
   • try siva unpack using Go implementation

If it works - log an issue in https://github.com/src-d/siva-java

2. Apache Spark can't have partition more than 2G
   Will update this later today

from gemini.

1. I have created an issue already. Because even if it's by-design it should be documented.

But here is test:

$ ls -lah
-rw-r--r--. 1 root root 4.6G Jan  5 09:48 ec644e00e3cab40629bc32562269c011ec2a6b14.siva
$ siva unpack ec644e00e3cab40629bc32562269c011ec2a6b14.siva
$ ls
HEAD  config  ec644e00e3cab40629bc32562269c011ec2a6b14.siva  go  objects  refs

Big files are available in /apps/borges/too-big if anybody else need them.

from gemini.

Nice, why not linking the issue here?

from gemini.

siva-java Issue was fixed src-d/siva-java#18 (comment) and new v0.1.3 was released + a new engine version that includes it.

from gemini.

with the last engine processing works

from gemini.