Comments (7)
Good point haha. I honestly believed I did, but no. Here you go: src-d/siva-java#18
from gemini.
Siva reader uses MappedByteBuffer:
https://github.com/src-d/siva-java/blob/master/src/main/java/tech/sourced/siva/SivaReader.java#L43
And there is a limit in jdk https://github.com/openjdk-mirror/jdk7u-jdk/blob/master/src/share/classes/sun/nio/ch/FileChannelImpl.java#L788
Then there is an issue in Spark, it can't have partition more than 2G:
https://issues.apache.org/jira/browse/SPARK-6235
The issue is open since 2015 year but no much progress since then.
We can't change it.
from gemini.
2 issues here:
-
siva-java fails to read .siva files > 2Gb
To understand, if that is by-design, or is it a 🐛 I would suggest
- find a repository that results in > 2Gb .siva file
borges pack
it locally- try
siva unpack
using Go implementation
If it works - log an issue in https://github.com/src-d/siva-java
- Apache Spark can't have partition more than 2G
Will update this later today
from gemini.
- I have created an issue already. Because even if it's by-design it should be documented.
But here is test:
$ ls -lah
-rw-r--r--. 1 root root 4.6G Jan 5 09:48 ec644e00e3cab40629bc32562269c011ec2a6b14.siva
$ siva unpack ec644e00e3cab40629bc32562269c011ec2a6b14.siva
$ ls
HEAD config ec644e00e3cab40629bc32562269c011ec2a6b14.siva go objects refs
Big files are available in /apps/borges/too-big
if anybody else need them.
from gemini.
Nice, why not linking the issue here?
from gemini.
siva-java Issue was fixed src-d/siva-java#18 (comment) and new v0.1.3 was released + a new engine version that includes it.
from gemini.
with the last engine processing works
from gemini.
Related Issues (20)
- Document ability to work with distrubited date sources
- Move docker build/upload in travis to separate stage
- Incorrect listing of repositories for bare/standard mode
- Make some constants configurable
- Remove feature-extractor service
- Update user documentation how to run gemini
- Update development documentation
- Investigate if it is possible to run igraph from scala
- Create a script for fixtures update
- Document how to run subsets of tests
- Setup travis to re-run tests weekly
- Update dependencies
- Migrate to gitbase-spark-connector
- Remove unused code
- Gemini can't detect similar C# files
- Create a doc with list of improvements HOT 2
- All log messages should go to stderr
- AbstractMethodError after start HOT 1
- How to generate a repo's siva file? HOT 1
- Running with docker compose fails connecting to Cassandra HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from gemini.