aw / picolisp-json Goto Github PK

I noticed when a JSON string contains a caret ^ character next to another character, the parser will convert it incorrectly.

Incorrect:

# test.json
{"test":"x^2"}

# pil +
: (decode "test.json" T)
-> (("test" . "x^R"))

Correct:

# test.json
{"test":"x^2"}

# pil +
: (decode "test.json" T)
-> (("test" . "x\^2"))

The character sequences \\ and \" are not being encoded according to the JSON spec.

I discovered a bug in the json.l parser where it is unable to parse an empty array containing an empty string, ex:

: (load "json.l")
-> json-encode
: (println (json-decode "{\"myarray\":[\"\"],\"test\":12345}"))
(("myarray" T))
-> (("myarray" T))

Expected result:

: (load "json.l")   
-> json-encode
: (println (json-decode "{\"myarray\":[\"\"],\"test\":12345}"))
(("myarray" T) ("test" . 12345))
-> (("myarray" T) ("test" . 12345))

It appears a string which already contains double-quotes will not output valid JSON.

This would need to be tested for keys as well..

There's a bug which prevents null and false values from being encoded correctly when generating a JSON file/string.

The strings end up being encoded to an empty object {} instead, because they are returned as NIL.

JSON strings which contain multiple empty arrays or empty objects aren't parsed correctly, ex:

{
	"test": [{}, {}, {},
		[],
		[]
	]
}

Although it's valid JSON, this library throws up.

From the ECMA 404 specification, which this library should adhere to:

A number is a sequence of decimal digits with no superfluous leading zero.

The following JSON string:

{"Numbers cannot have leading zeroes": 013}

is successfully parsed to:

(("Numbers cannot have leading zeroes" . 13))

Technically it should reject the JSON as non-compliant, rather than strip the leading zeros..

The encoder doesn't handle special control characters due to native functionality of (str) in PicoLisp:

The combination of a backslash followed by 'n', 'r' or 't' is replaced with newline ("^J"), return ("^M") or TAB ("^I"), respectively.

These characters should be converted back to their original \n\r\t form when encoded. Same applies to \b and \f which are currently encoded as ^H and ^L respectively.

Encoding true, false, null is incorrect. Examples:

correct

: (encode '(("test" . true)))
-> "{\"test\":true}"

incorrect

: (encode '(("test" . "true")))
-> "{\"test\":true}"

expected

: (encode '(("test" . "true")))
-> "{\"test\":\"true\"}"

Tests should be done with pointer equality to ensure true = true and "true" = "true"

Note: Decoding is fine.

The unicode control characters 0x01 to 0x1F are not being escaped and thus not appearing in the JSON output.

The value 0x00 is NIL, so we ignore it.

Control characters between 0x01 and 0x1F, excluding the special control characters from #11 should be encoded as \u0001 to \u001F.

The following happens with PL 15.11.0 (ubuntu 16.04 global install) and PL 17.11.19 (local install):

erik@thinkster2:~/code/pl/picolisp-json$ pil json.l +

: (json~encode '(("Hello" . "World")))
-> "{\"Hello\":\"World\"}"

: (json~decode @)                   
!? ((any (pack "json-" "Type" "-check")) "Name")   # from 'link-generic', see below
json-object-check -- Undefined
?
                
: (symbols 'json)
-> pico

json: (encode '(("Hello" . "World")))
-> "{\"Hello\":\"World\"}"

json: (decode @)                     
-> (("Hello" . "World"))

The line ((any (pack "json-" Type" "-check")) Name) seems to build a symbol (in this case json-object-check) to be called as a function on Name. When called from outside the the json namespace, it still returns json-object-check, which is local to the json namespace and thus undefined.

The following is a quick fix that (seems) to work on my local copy:

erik@thinkster2:~/code/pl/picolisp-json$ git diff json.l
diff --git a/json.l b/json.l
index c25f792..13697ea 100644
--- a/json.l
+++ b/json.l
@@ -79,7 +79,7 @@
 [de link-generic (Type Bracket Iterator)
   (let Name (car *Json)
     (when Name
-          ((any (pack "json-" Type "-check")) Name)
+          ((any (if (= (symbols) 'json) (pack "json-" Type "-check") (pack "json~json-" Type "-check")) Name)
           (unless (= Bracket Name) (eval Iterator)) )
 
     (if (= Bracket (car *Json))

But this breaks all the unit tests, as PIL_NAMESPACES is set to false for testing:

erik@thinkster2:~/code/pl/picolisp-json$ make check 
PIL_NAMESPACES=false ./test.l
# MODULE_INFO redefined

  Testing JSON encoder/decoder for PicoLisp

"Unmatched JSON brackets '{'"
[test_json.l:96] json -- Bad symbol namespace
? 

If the following strings are supplied, the JSON parser fails to decode correctly:

• \u005c: \
• \u005e: ^
• \u0022: "

This is caused by the PicoLisp reader, but we can easily workaround this by escaping those characters during conversion.