michel-kraemer / actson Goto Github PK
View Code? Open in Web Editor NEWA reactive (or non-blocking, or asynchronous) JSON parser
License: MIT License
A reactive (or non-blocking, or asynchronous) JSON parser
License: MIT License
I have a use case parsing a very big JSON document that's structured like
{
"fieldName":"35cdf2614146t14314[...]" /* many megabytes of base-64 data */
}
It seems that currently the StringBuilder currentValue
will just keep growing unbounded as a string value rolls in.
In order to be bounded, there could be a maximumStringValueLength
setting that puts an upper limit on the memory usage. But even better would be to allow (possibly by another flag) for the string value to be separated into multiple events, so it can be handled in a streaming way.
It'd need a new VALUE_PARTIAL_STRING
event though, or sth like that.
See test suite here:
https://github.com/nst/JSONTestSuite
And the article Parsing JSON is a Minefield by Nicolas Seriot (@nst):
http://seriot.ch/parsing_json.html
Since Actson is based on JSON Checker it will probably suffer from the issues described in section 4.8.
Unicode symbols represented as UTF-8 in strings are parsed as expected. I also see that parser has state ES
(escape). But escapes of Unicode code-points are not converted to respective code points.
For example source text "\"\\u00e9\"".getBytes()
is parsed as single token "\\u00e9"
instead of "é"
.
Here's the test ba58753
I am thinking about getting rid of other validating legacy of thins code. Procedures that use feeder look cumbersome (see these loops in d.u.a.JsonParserTest#parseFail, for example). So removing feeder from parser altogether should simplify code and it's usage. Looking at d.u.a.JsonParser#nextEvent I now think that whole point of having feeder is the ability to detect premature end of input and return EOF or ERROR events appropriately. Application that is only interested in parsing still can trivially infer these conditions anyway. Here's changes that I propose.
Change API of parser to
public class JsonParser {
public JsonParser(int depth) {...} // Initial state, only constructor
public int getMaxDepth() {...} // Keep this
public int nextEvent(char nextChar) {...} // Change state with next input text's character, return event.
// All else would be non public or removed
}
This effectively reduces parser to be state machine only. This keeps input buffering out of parser so clients are free to use whatever they want without implementing interface.
Replace event EOF with, say, END which is emitted when current JSON structure is completely parsed. This happens when we have already parsed a scalar value or end of nested structure is popped and stack is empty.
Here are 2 options: after issuing END all subsequent calls would produce ERROR which is similar to current behavior, or keep parsing as we're at the beginning. In case when sequence of multiple JSON documents is expected in input first option would require to re-create parser to parse new piece, later case allow to reuse it and just interpret END event if necessary. Since I am interested in this use-case I'd prefer to keep parsing after END.
setDepth
setter should be removed since it is not clear what to do if we are in the middle of parsing, say, at depth 100 and this setting is changed to 50.
The parser usage example would be like:
final JsonParser jsonParser = new JsonParser(100);
final CharSequence jsonSource = "{\"abc\":123}";
int i = 0;
int event;
while (i < jsonSource.length()) {
event = jsonParser.nextEvent(jsonSource.charAt(i++));
if (event != NEED_MORE_INPUT) {
// One can handle END event explicitly here, e.g. break loop
processEvent(event);
}
}
if (event == NEED_MORE_INPUT) {
throw new RuntimeError("Incomplete JSON input.");
}
This is a feature request rather than a defect.
I'm using actson in a netty 4.1.17 environment to stream json content that is being received from a chunked http transfer.
Using the JsonFeeder interface it is not possible to feed data from a netty ByteBuf without copying it to a byte array first. It is, however, possible to wrap the ByteBuf as a standard java.nio ByteBuffer. Would it be reasonable to add a feed method that takes the ByteBuffer?
I've experimented with implementing my own feeder but I would like to avoid duplicating DefaultJsonFeeder's charset decoding. An alternative approach would be to factor the charset decoding into a separate class.
As as side note, the feeder interface could reasonably be split into two interfaces (one that the parsers uses to consume input and one that application code uses to provide input). I suspect this change would be too disruptive to existing users.
If any of these ideas sound reasonable I am happy to help make a pull request.
I've encountered a weird issue. In the example I've attached in line 18 is a unicode escape. This seems to parse fine. However, the parser now breaks in line 49. If I remove line 18, line 49 (then 48) is properly read. Also, if I remove the unicode escape from that line it all works out fine.
To clarify: I'm using the JsonParser stream parser with default encoding. The file is also UTF8 encoded but still has these escapes.
I am building an application that consumes large files of structured data and does some processing based on the content.
I'm using the Aalto streaming parser for XML files and this looks like the most similar option for JSON.
One of the requirements for my project is that it must be able to handle large string values in the JSON, which may be too large to hold in memory. I'm wondering if there is any way I can use this parser in its current form to handle this case since as far as I can tell it only emits a VALUE_STRING JsonEvent once the parser has seen the closing "
.
Currently JsonFeeder
lacks the ability to directly feed off read-only ByteBuffer
objects. This has the disadvantage of needing another copy operation for frameworks that offer input bytes as ByteBuffer
(e.g. akka's ByteString
and google protobuf's ByteString
do this).
JsonFeeder
should be extended to read from byte buffers as well. On the implementation side, I see two possible implementations:
fillBuffer
is called (no copying)Actually, glancing at the code, is (2) even handled currently?
I'd be happy to do a PR.
Hi
How does the performance compare with existing libraries such as Jackson? I consider using it for network application in which JSON messages arrive in chunks.
It would be useful if when receiving the JsonEvent.ERROR
event that there was a property set that could explain the reason why the parse failed. Looking through the JsonParser
source this seems like it would be simple enough to do since most error conditions are the result of an unexpected token.
I'd be happy to tackle this myself.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.