What types of JSON serialization do we support in params? about carbond HOT 1 CLOSED

Hey guys-

I had to write this out to think about what we should do. (Greg and I have been discussing this in the office over the past few days).

I've been working on Carbon.io's parameter parser and run into a thorny issue.

Here is a description of the problem, and what I propose as the solution. As users of Carbon.io let me know if this makes sense:

Problem

The world has somehow produced what I am calling DJSON (De-normalized JSON) in order to serialize JSON as HTTP query parameters instead of just encoding JSON parameter values as strings.

For example, if you have a query parameter named p with an object value of { "x": 1 } the resulting query string will be:

?p[x]=1

The problem is that if the value of p is { "x": '1' } the query string is also:

?p[x]=1

This means that there is no way to distinguish between numbers and strings (and booleans) with DJSON. Any DJSON parser must choose one way to interpret the value, which in the case of most implementations we know of (like Node's Querystring or Qs libraries), will be as a string.

This really sucks. The whole point of DJSON was to presumably provide a serialization mechanism for JSON. Any serialization scheme must have the property that deserialize(serialize(x)) === x for all possible values of x. This does not hold for DJSON.

Unfortunately however, we have to deal with the reality that there are many clients out there that produce DJSON despite the fact that they could simply serialize the JSON as a string and sidestep the whole problem.

Solution: We must support four use cases

(1) Client serializes to DJSON (Denormalized JSON), No schema defined on server
---> PROBLEM: Parameter can result in an object with possibly incorrect types
---> SOLUTION: Client should serialize as JSON string and define schema on server

(2) Client serializes to DJSON, Schema defined on server
---> PROBLEM: Parameter may not validate schema due to incorrect types OR may validate schema (because the schema is something course grained) but still have incorrect types
---> SOLUTION: We coerce OR client should serialize as JSON string. Coerce will usually work if there is enough schema defined. If schema is course (like type: object) then we cannot coerce and client will need to serialize as string.

(3) Client serializes to encoded JSON string, No schema defined on server
---> PROBLEM: Parameter is a string, and developer does not expect that
---> SOLUTION: Define schema on server

(4) Client serializes to encoded JSON string, Schema defined on server
---> No problems

General stance then would be that the server should always define a schema if it is a public API or has little control over the client. It is best practice anyway so this should not be controversial.

Then the tasks comes to educating users (and them educating their customers) that it is better to pass JSON as strings if they are using a library the serializes to DJSON. This means either:

(A) They use same library but produce { "sort": '{"x": 1}' } instead of { "sort": {"x" : 1}}
OR
(B) Use a Carbon Client library which will do the right thing for them

Suggestion to DJSON folks

Serialize strings with quotes. Then call JSON.parse on the leaves as you reconstruct the JSON. The datum '1' will evaluate to the number 1, and the datum '"1"' will evaluate to the string "1". Problem solved, and deserialize(serialize(x)) === x.

Plan for us

Document this clearly so our Carbon.io users have a smooth experience. Is the set of PROBLEM/SOLUTION above comprehensive? Are the solutions acceptable? Any other possible paths we might take?

from carbond.