Comments (6)
Thanks for the report @mmuurr
We came upon this recently, see ropensci/geojson#36
The answer is essentially that you're pushing too much data in at once, so try to push in smaller chunks if possible. Is it possible in your case?
@jeroen With this example that 100K length JSON works fine with jq on the cli, so is there anything we can do change this? If not, maybe we can help users split up json into chunks and then re-combine. Would work if it's like in the example above where each element in a vector is valid JSON, but not so easy otherwise
from jqr.
@sckott, yeah I'm accommodating for now by chunking inputs (e.g. via readr::read_lines_chunked
), but I thought I'd just raise the issue for awareness :-)
from jqr.
glad you can break it up. we'll see what Jeroen says.
from jqr.
Hi there, just a polite re-surfacing of this issue, which I've run into again. I think breaking up long JSON (character) vectors into chunks works just great and designing a simple wrapper to do so that then recombines the jqr results is indeed relatively easy. I'm wondering if there's:
- Any guidance on what that appropriate chunk size would be in number of strings (i.e. length of vector) and/or
- If the chunking should be determined by total byte size of the chunks, which adds some (albeit small) complexity to the wrappers.
Also should such a wrapper be integrated directly into jqr? (If so, I'd be happy to take a first stab at that wrapper and create a PR, though I'll pass on that effort if y'all don't believe it should be part of the package).
And if no chunking wrapper built-in, should jqr catch that specific type of error and update the user with 'advice' (i.e. "hey user, try chunking")?
from jqr.
Thanks @mmuurr - sorry for the delay on this.
That makes sense that it's 10K, since
https://github.com/stedolan/jq/blob/9b51a0852a0f91fbc987f5f2b302ff65e22f6399/src/parser.c#L1692 via jqlang/jq#1054 and jqlang/jq#1041
I think a wrapper belongs here in the package.
byte size does seem like it would be more appropriate.
can you send a PR and we can discuss from there
from jqr.
I am also experiencing this error when I feed more than 50k json strings into jq to process. I can chunk the data, of course, but is a bit disruptive in my example.
from jqr.
Related Issues (20)
- Look into ndjson-cli HOT 1
- make it clear/possible how to use modules
- Fix flags HOT 3
- Encoding with jqr::jq HOT 7
- Unhandled exception HOT 4
- sysreqs HOT 2
- Release 1.1.0 or note that build_object is only available if installed from GitHub HOT 6
- cran warning: strncpy HOT 1
- libjq was not found on Linux systems HOT 2
- NA handling HOT 4
- jq() paste bare numbers together HOT 8
- Error message suppressed and incorrect result returned HOT 4
- Better error message in check1() HOT 2
- jqr in CentOS8 : * rpm: jq-devel (Fedora, EPEL) EXist? HOT 3
- Passing --raw-output jq flag to enable json -> csv conversion to be "pushed down" to jq and enabling jqr to picking up nonjson results HOT 4
- Change to new cran checks badge URL HOT 1
- How to change values in the high level
- configure does not work correctly
- JSON field missing HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from jqr.