iskra / jsonx Goto Github PK

View Code? Open in Web Editor NEW

92.0 92.0 32.0 343 KB

JSONX is an Erlang library for efficient decode and encode JSON, written in C.

License: Other

C 66.05% Erlang 33.95%

jsonx's People

Contributors

Stargazers

Watchers

jsonx's Issues

Segmentation fault under high load on Solaris

It seems that there is a bug which exhibits itself under heavy load. In my case this means parsing and encoding huge JSON data ~50mb in parallel by ~10 erlang processes. Oh yes that was a Solaris OS, it might be the reason of a bug.

Anyway it might be interesting for you to see the following (from core dump):

 # mdb core
Loading modules: [ libc.so.1 libuutil.so.1 ld.so.1 ]
> ::stack
libc_psr.so.1`memcpy+0x58(f977be50, 1e46132e, f977bd78, 4000013, 224000, 44dcc)
jsonx.so`match_string+0x1c(f977be50, 1e46132e, f977bd78, f977ba54, 1e461344, 2)
jsonx.so`match_pair+0x60(f977be50, 1e461342, f977bd78, f977bd94, 640bfffb, 2c)
jsonx.so`match_proplist+0x158(f977be50, 1e460de9, f977bd78, f977bb5c, 1e460fb4, 1e460fb4)
jsonx.so`match_tuple+0xd0(f977be50, 1e460fb2, f977bd78, 0, f977bd78, 62080028)
jsonx.so`match_term+0x154(f977be50, 1e460fb2, f977bd78, f977bd94, 640bfd79, 2c)
jsonx.so`match_list+0x1ac(f977be50, 1a800019, f977bd78, 0, e3279240, 419bc)
jsonx.so`match_term+0xec(f977be50, 1a800019, f977bd78, 7cafc, 0, f977bd78)
jsonx.so`encode_nif+0x174(f977be50, 1, fce86180, 3e7, f80e3b98, 27a84388)
process_main+0x6c4c(4, 4, fa216410, 8, fe435788, 1)
sched_thread_func+0x248(fca7f400, f9d00080, f9d00080, 0, f9d00880, 3)
thr_wrapper+0x70(ffbff128, f977c000, 0, 0, 1a0040, 0)
libc.so.1`_lwp_start(0, 0, 0, 0, 0, 0)

Extra/missing field in JSON fails even with {format, proplist}

Note: this very long issue contains three different examples, that's why it's so long :)

I'm working with Neo4J REST API. The JSON they return for some of the objects may contain optional fields.

Erlang R16B02

Example 1. Extra field

For example, this is JSON for a node which is not in an index:

{
  "extensions" : {
  },
  "paged_traverse" : "http://localhost:7474/db/data/node/376/paged/traverse/{returnType}{?pageSize,leaseTime}",
  "outgoing_relationships" : "http://localhost:7474/db/data/node/376/relationships/out",
  "traverse" : "http://localhost:7474/db/data/node/376/traverse/{returnType}",
  "all_typed_relationships" : "http://localhost:7474/db/data/node/376/relationships/all/{-list|&|types}",
  "property" : "http://localhost:7474/db/data/node/376/properties/{key}",
  "all_relationships" : "http://localhost:7474/db/data/node/376/relationships/all",
  "self" : "http://localhost:7474/db/data/node/376",
  "outgoing_typed_relationships" : "http://localhost:7474/db/data/node/376/relationships/out/{-list|&|types}",
  "properties" : "http://localhost:7474/db/data/node/376/properties",
  "incoming_relationships" : "http://localhost:7474/db/data/node/376/relationships/in",
  "incoming_typed_relationships" : "http://localhost:7474/db/data/node/376/relationships/in/{-list|&|types}",
  "create_relationship" : "http://localhost:7474/db/data/node/376/relationships",
  "data" : {
  }
}

And this is JSON for a node that has been added to an index:

{
  "extensions" : {
  },
  "paged_traverse" : "http://localhost:7474/db/data/node/8/paged/traverse/{returnType}{?pageSize,leaseTime}",
  "outgoing_relationships" : "http://localhost:7474/db/data/node/8/relationships/out",
  "traverse" : "http://localhost:7474/db/data/node/8/traverse/{returnType}",
  "all_typed_relationships" : "http://localhost:7474/db/data/node/8/relationships/all/{-list|&|types}",
  "property" : "http://localhost:7474/db/data/node/8/properties/{key}",
  "all_relationships" : "http://localhost:7474/db/data/node/8/relationships/all",
  "self" : "http://localhost:7474/db/data/node/8",
  "outgoing_typed_relationships" : "http://localhost:7474/db/data/node/8/relationships/out/{-list|&|types}",
  "properties" : "http://localhost:7474/db/data/node/8/properties",
  "incoming_relationships" : "http://localhost:7474/db/data/node/8/relationships/in",
  "incoming_typed_relationships" : "http://localhost:7474/db/data/node/8/relationships/in/{-list|&|types}",
  "create_relationship" : "http://localhost:7474/db/data/node/8/relationships",
  "data" : {
  },
  "indexed" : "http://localhost:7474/db/data/index/node/favorites/some-key/some%20value/8"
}

The only difference is the "indexed" field.

Given this record:

-record(neo4j_node, { extensions
                    , paged_traverse
                    , outgoing_relationships
                    , traverse
                    , all_typed_relationships
                    , property
                    , all_relationships
                    , self
                    , outgoing_typed_relationships
                    , properties
                    , incoming_relationships
                    , incoming_typed_relationships
                    , create_relationship
                    , data
                    , labels
                    }
      ).

and

> Decoder = jsonx:decoder( [{neo4j_node, record_info(fields, neo4j_node)}]
                         , [{format, proplist}]).

Decoding the first JSON will yield a correct record:

#neo4j_node{
    extensions = [],
    paged_traverse =
        <<"http://localhost:7474/db/data/node/189/paged/traverse/{returnType}{?pageSize,leaseTime}">>,
    outgoing_relationships =
        <<"http://localhost:7474/db/data/node/189/relationships/out">>,
    traverse =
        <<"http://localhost:7474/db/data/node/189/traverse/{returnType}">>,
    all_typed_relationships =
        <<"http://localhost:7474/db/data/node/189/relationships/all/{-list|&|types}">>,
    property =
        <<"http://localhost:7474/db/data/node/189/properties/{key}">>,
    all_relationships =
        <<"http://localhost:7474/db/data/node/189/relationships/all">>,
    self = <<"http://localhost:7474/db/data/node/189">>,
    outgoing_typed_relationships =
        <<"http://localhost:7474/db/data/node/189/relationships/out/{-list|&|types}">>,
    properties =
        <<"http://localhost:7474/db/data/node/189/properties">>,
    incoming_relationships =
        <<"http://localhost:7474/db/data/node/189/relationships/in">>,
    incoming_typed_relationships =
        <<"http://localhost:7474/db/data/node/189/relationships/in/{-list|&|types}">>,
    create_relationship =
        <<"http://localhost:7474/db/data/node/189/relationships">>,
    data = [{<<"born">>,1940},{<<"name">>,<<"Al Pacino">>}],
    labels =
        <<"http://localhost:7474/db/data/node/189/labels">>}

however, it will fail on the second JSON:

{error,invalid_json,1}

Even though I expected it to return a proplist (as it does for some other structures).

If I add "indexed" to record definition, decoder will fail with {error,invalid_json,1} for the first JSON and not for the second.

Example 2. Missing field

The service root may or may not contain a field called "reference_node":

{
  "extensions" : {
  },
  "node" : "http://localhost:7474/db/data/node",
  "reference_node" : "http://localhost:7474/db/data/node/371",
  "node_index" : "http://localhost:7474/db/data/index/node",
  "relationship_index" : "http://localhost:7474/db/data/index/relationship",
  "extensions_info" : "http://localhost:7474/db/data/ext",
  "relationship_types" : "http://localhost:7474/db/data/relationship/types",
  "batch" : "http://localhost:7474/db/data/batch",
  "cypher" : "http://localhost:7474/db/data/cypher",
  "neo4j_version" : "1.9.5"
}

Given this record definition:

-record(neo4j_root, { extensions
                    , node
                    , reference_node
                    , node_index
                    , relationship_index
                    , extensions_info
                    , relationship_types
                    , batch
                    , cypher
                    , transaction
                    , neo4j_version
                    }
       ).

and

Decoder = jsonx:decoder( [{neo4j_root, record_info(fields, neo4j_root)}]
                       , [{format, proplist}]).

this will fail with {error,invalid_json,1} if reference_node isn't in the JSON.

Example 3. Everything works as expected

For some other JSON this works as expected. JSON for indices can be either

{
  "template" : "http://localhost:7474/db/data/index/node/favorites/{key}/{value}"
}

{
  "template" : "http://localhost:7474/db/data/index/node/fulltext/{key}/{value}",
  "type" : "fulltext",
  "provider" : "lucene"
}

Given

-record(neo4j_index, { template
                     , type
                     , provider
                     }
       ).
...
Decoder = jsonx:decoder( [{neo4j_index, record_info(fields, neo4j_index)}]
                       , [{format, proplist}]).

the first JSON will be decoded as

[{<<"template">>, <<"http://localhost:7474/db/data/index/node/fulltext/{key}/{value}">>}]

and the second one as

#neo4j_index{template = <<"http://localhost:7474/db/data/index/node/aa/{key}/{value}">>,
               type = <<"exact">>,provider = <<"lucene">>}

That is, as expected.

I have no idea why this happens :)

Library load-call unsuccessful

This error does not occur every time, but it occurs.
Don't know how to reproduce it.

    =ERROR REPORT==== 15-Apr-2013::06:31:34 ===
    The on_load function for module jsonx returned {error,
                                            {load,
                                             "Library load-call unsuccessful."}}

Error in output.

-module(json_t).

-record(json_t, {
    name :: binary(),
    age = 0 :: pos_integer(),
    spouse :: #json_t{}
}).

-record(json_t2, {
  name :: binary(),
  age = 0 :: pos_integer(),
  spouse :: #json_t{}
}).

encoder1() ->
    jsonx:encoder([{json_t, record_info(fields, json_t)},
                             {json_t2, record_info(fields, json_t2)} ], 
                             [{ignore, [undefined]}]).

a() ->
  R = #json_t2{ name = <<"John">>, age = 32, spouse = [#json_t{}] },
  Encoder = encoder1(),
  Res = Encoder(R),
  io:format("~p ~n",[Res]).

RESULT:

<<"{"name": "John","age": 32,"spouse": [{,"age": 0}]}">>

If first field in record == undefined.

no_match should point to erroneous tuple

If one tries to encode large proplist, it is difficult to find property with error because jsonx always points to first tuple:
3> jsonx:encode([{<<"b">>, false},{<<"a">>, 999999999999999999999999999999999999999999999999999}]).
{no_match,{<<"b">>,false}}

Pointing to tuple with error makes error finding easier.

Cannot parse string with \ud83d

Returns {error, invalid_string, _} when trying to parse string with \ud83d

Work with big numbers

Just found jsonx can't work with really big numbers:

jsonx:encode([{<<"a">>, 999999999999999999999999999999999999999999999999999999}]).
{no_match,{<<"a">>,
999999999999999999999999999999999999999999999999999999}}

Is there any ways to encode big numbers to valid json?

Error when I compile with rebar3

When I use rebar3 to compile jsonx,got error like this:

=ERROR REPORT==== 19-Apr-2017::11:36:37 ===
Error in process <0.207.0> with exit value: {{badmatch,{error,{load_failed,"Failed to load NIF library: '/home/dasudian/Proj/swdportal/deps/jsonx/_build/test/lib/jsonx/priv/jsonx.so: cannot open shared object file: No such file or directory'"}}},[{jsonx...

Everything is ok when I compile with rebar...
Is there any advice?Thanks

Failed compilation on MSVC

There are a few issues when compiling on MSVC compiler. Specifically:

No support for inline keyword in ISO C90. It can be fixed for MSVC by

#ifdef _MSC_VER
#define inline __inline
#endif

Illegal by C standard void pointer arithmetic (e.g. jsonx.h:79, jsonx.h:84). GCC allows this as an extension, but MSVC does not.

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.