[tm_json.h] Comments interrupt parsing about tm HOT 4 CLOSED

Thank you for reporting the issue. While testing I also uncovered more issues with how comments were handeled. The newest version should properly support comments between all json tokens. I had to make some structural changes to the parser and added a new token type, which is why some more rigorous tests were also added. The parser is now being tested against JSONTestSuite and json5-tests.

Note that when using JSON_READER_JSON5 it is necessary to use jsonAllocateDocumentEx to make use of all features of Json5. The regular jsonAllocateDocument only supports a handful of flags for simplicity. There are also some caveats with Json5 currently regarding whitespace (json5 allows many unicode codepoints to be regarded as whitespace, which this parser rejects) or unicode letters in unquoted property names (which this parser rejects). Unicode is not a problem in general, just in these two cases.

from tm.

"Parsing json is a minefield" says the guy. I thought parsing XML was a minefield. This stuff is no joke, and I'm amazed at how fast you delivered a fix. Thanks!

I'm not totally blameless in this after all. jsonAllocateDocumentEx. Yes, I forgot. Now, I'm not sure if this one is just me, but with the newest version of tm_json.h when I change line 19 of my test case to this:

jsonAllocateDocumentEx( text, num_chars, JSON_READER_JSON5 );

I get this:

test1.json is 33 characters long
test2.json is 30 characters long
JSON ERROR: Unexpected Token at line 7:0

Line 7 of test2.json does not exist, and therere is no error for test1.json. One of those weird whitespace errors or something to do with //?

from tm.

Nice catch. Seems in my rewrite of whitespace handling I forgot to boundscheck. My testing didn't expose this since I am using c strings that are null terminated. Since your code uses an uninitialized buffer, it triggered the out of bounds read. I think I fixed it and the tests now copy strings into a new non nullterminated buffer with invalid chars at the end to trigger errors on out of bounds reads.

from tm.

\xfe invalidates it. My own code is apparently not the most pedantic, so this works for me! No errors in my test case.

from tm.