protection stack overflow errors on medium(?)-sized inputs about jqr HOT 6 OPEN

Thanks for the report @mmuurr

We came upon this recently, see ropensci/geojson#36

The answer is essentially that you're pushing too much data in at once, so try to push in smaller chunks if possible. Is it possible in your case?

@jeroen With this example that 100K length JSON works fine with jq on the cli, so is there anything we can do change this? If not, maybe we can help users split up json into chunks and then re-combine. Would work if it's like in the example above where each element in a vector is valid JSON, but not so easy otherwise

from jqr.

@sckott, yeah I'm accommodating for now by chunking inputs (e.g. via readr::read_lines_chunked), but I thought I'd just raise the issue for awareness :-)

from jqr.

glad you can break it up. we'll see what Jeroen says.

from jqr.

Hi there, just a polite re-surfacing of this issue, which I've run into again. I think breaking up long JSON (character) vectors into chunks works just great and designing a simple wrapper to do so that then recombines the jqr results is indeed relatively easy. I'm wondering if there's:

1. Any guidance on what that appropriate chunk size would be in number of strings (i.e. length of vector) and/or
2. If the chunking should be determined by total byte size of the chunks, which adds some (albeit small) complexity to the wrappers.

Also should such a wrapper be integrated directly into jqr? (If so, I'd be happy to take a first stab at that wrapper and create a PR, though I'll pass on that effort if y'all don't believe it should be part of the package).

And if no chunking wrapper built-in, should jqr catch that specific type of error and update the user with 'advice' (i.e. "hey user, try chunking")?

from jqr.

Thanks @mmuurr - sorry for the delay on this.

That makes sense that it's 10K, since
https://github.com/stedolan/jq/blob/9b51a0852a0f91fbc987f5f2b302ff65e22f6399/src/parser.c#L1692 via jqlang/jq#1054 and jqlang/jq#1041

I think a wrapper belongs here in the package.

byte size does seem like it would be more appropriate.

can you send a PR and we can discuss from there

from jqr.

I am also experiencing this error when I feed more than 50k json strings into jq to process. I can chunk the data, of course, but is a bit disruptive in my example.

from jqr.