It would be nice for jiffy to support Elixir, namely the way null/nil is handled. Currently it's not ideal:

iex(1)> :jiffy.encode %{results: nil}                        
"{\"results\":\"nil\"}"
iex(2)> :jiffy.decode "{\"results\":null}", [:return_maps]   
%{"results" => :null}

When encoding, a nil atom should be turned into a JSON null, not a "nil" string as in the current implementation. Likewise, decoding a JSON null should result in a nil atom, not null. I'm not sure about the best way to handle this, but perhaps some option like use_nil_for_null could be added so jiffy can be painlessly used from Elixir :)

For example:

% Encode Erlang date tuples into ISO8601 dates
encode_handler({Y, M, D}) when is_integer(Y); is_integer(M); is_integer(D) ->
    Fmt = "~4.10.0B-~2.10.0B-~2.10.0B",
    io_lib:format(Fmt, [Y, M, D]);
encode_handler(Other) ->
    Other.

Json = jiffy:encode([{date, {2012,1,1}}], [{handler, fun encode_handler/1}]).

In README:
1.25 -> 1.25 -> 1.24

I have a case where I build some dynamic JSON returned for chunked encoding in HTTP. I return some JSON in each chunk, but I want the whole object in the end to be valid JSON.

Example:

{
    "chunks": [
        {"data": 1},
        {"data": 2},
        {"data": 3},
        {"data": 4},
        {"data": 5}
    ],
    "other": "something",
    "more": "extra"
}

So, I start out by sending the binary [<<"{\"chunks\":[">>, jiffy:encode(#{data => 1})] and then periodically I send [<<",">>, jiffy:encode(#{data => N}]. In the end, I want to send #{other => something, more => extra} but without the enclosing object brackets.

Would it be possible to add an option like keys_only or something? Example:

1> jiffy:encode(#{other => something, more => extra}).
<<"{\"other\":\"something\",\"more\":\"extra\"}">>
2> jiffy:encode(#{other => something, more => extra}, [keys_only]).
<<"\"other\":\"something\",\"more\":\"extra\"">>

This could be used to efficiently build and concatenate sub parts of JSON documents.

Looks like error contributed in e9a102a.
https://github.com/davisp/jiffy/blob/master/c_src/util.c#L47

    cmd: cc -c  -g -Wall -Werror -O3 -fno-strict-aliasing -g -Wall -fPIC  -I/usr/lib/erlang/lib/erl_interface-3.7.16/include -I/usr/lib/erlang/erts-6.0/include   c_src/util.c -o c_src/util.o
Compiling c_src/util.c
c_src/util.c: В функции «get_bytes_per_iter»:
c_src/util.c:47:5: ошибка: в передаче аргумента 3 «enif_get_uint64»: несовместимый тип указателя [-Werror]
In file included from /usr/lib/erlang/erts-6.0/include/erl_nif.h:224:0,
                 from c_src/jiffy.h:7,
                 from c_src/util.c:4:
/usr/lib/erlang/erts-6.0/include/erl_nif_api_funcs.h:134:1: замечание: expected «ErlNifUInt64 *» but argument is of type «size_t *»

(sorry for russian, I don't know how to change locale for rebar)

Note, that, according to docs, 3'rd parameter of enif_get_uint64 should be ErlNifUInt64 *

seriy:jiffy/ (307c383) $ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 13.04
Release:    13.04
Codename:   raring
seriy:jiffy/ (307c383) $ uname -a
Linux seriy-ostrovok 3.8.0-35-generic #50-Ubuntu SMP Tue Dec 3 01:25:33 UTC 2013 i686 i686 i686 GNU/Linux
seriy:jiffy/ (307c383) $ rebar -V
rebar 2.1.0-pre R15B01 20130325_132426 git 2.1.0-pre-46-g78fa8fc
seriy:jiffy/ (307c383) $ erl
Erlang/OTP 17 [erts-6.0] [source-07b8f44] [smp:8:8] [async-threads:10] [hipe] [kernel-poll:false]

I'd like to safe Erlang terms persistently. They are to be encoded by jiffy some time afterwards. To prevent encoding errors, I'd like to check the Erlang terms, before I save them without actually encoding them.

The signature would basically look like this:
check(EJSON) -> ok|{err, Reason}

Would that feature be useful for anyone else?

Per JSON spec, The names within an object SHOULD be unique.. See COUCHDB-1294 for more discussion. Should jiffy return invalid_json for this case?

%% OK
(term@akai)16> jiffy:decode(<< "{\"foo\":\"bar\", \"few\":\"bar\"}" >>).
{[{<<"foo">>,<<"bar">>},{<<"few">>,<<"bar">>}]}

%% Would expect either invalid_json or possibly a new error, duplicate_name_in_object
(term@akai)17> jiffy:decode(<< "{\"foo\":\"bar\", \"foo\":\"bar\"}" >>).
{[{<<"foo">>,<<"bar">>},{<<"foo">>,<<"bar">>}]}

It would be interresting to have a way to return objects as just a proplists from time to time.

The encoding part is the most difficult but one way to do it woul be matching [{_Key, _Val} | _REst ], [{Key,Val}] and [{}] for empty objects.

What do you think?

Failed to encode {[{event, query_done}, {data, {id, aaa}}]}.

json_encode2_test() ->
    In = {[{event, query_done}, {data, {id, aaa}}]},
    JsonStr = jiffy:encode(In),
    ?assertEqual(<<"{\"event\":\"query_done\",\"data\":{\"id\":\"aaa\"}}">>, JsonStr).

Executed this test...

in function jiffy:encode/2 (src/jiffy.erl, line 34)
in call from websocket_handler_tests:json_encode2_test/0 (test/websocket_handler_tests.erl, line 11)
**throw:{error,invalid_ejson}

Nested object is not supported?

Is there any way you can add the undefined atom as having the same functionality as null. Erlang uses undefined within the standard lib and I think it makes things more consistent. I also do think that null in Json should translate back to the undefined atom. Or maybe you could change it to configure what the user wants null in Json to translate to in Erlang?

1> jiffy:encode(0.1).
<<"0.10000000000000000555">>

I believe this is due to the use of sprintf("%0.20g",...) in encoder.c

Perhaps Erlang's own facilities could be leveraged instead:

2> list_to_binary(hd(io_lib:format("~p", [0.1]))).
<<"0.1">>

Erlang R16B03-1
I have reproduced an issue when encoding big integers with jiffy. I tried version 13 and version 8.5. If you run this piece of code in your shell:

P2 = fun() ->
        spawn_link(fun() ->
            Parent = self(),
            [erlang:garbage_collect(Pid) || Pid <- erlang:processes()],
            Before = erlang:memory(binary),
            io:format("Before ~p~n", [Before div 1000]),
            Child = spawn_link(
                fun() ->
                    [jiffy:encode(1072502107250210725021072502) || _ <- lists:seq(1, 1000000)],
                    Parent ! {finished, self()}
                end),
            receive
                {finished, Child} ->
                    [erlang:garbage_collect(Pid) || Pid <- erlang:processes()],
                    After = erlang:memory(binary),
                    io:format("After ~p~n", [After div 1000]),
                    io:format("Diff ~p~n", [(After - Before) div 1000])
            end
        end)
        end.

And then execute P2() several times you can see that the memory is not released. This doesn't happen with small integers.

(boodah_dev@lingon)22> P2().
Before 21119807        
After 23207811         
Diff 2088003           
(boodah_dev@lingon)23> P2().
Before 23207809        
After 25295802         
Diff 2087992           
(boodah_dev@lingon)24> P2().
Before 25295791        
After 27383780         
Diff 2087988           

(boodah_dev@lingon)26> erlang:memory(binary).
27384865904
(boodah_dev@lingon)27> P2().                 
Before 27383796        
After 29471788         
Diff 2087991           

make check is failing on R15b :

enoitc:jiffy benoitc$ make check
prove -v test/*.t
test/001-yajl-tests.t ..

Current time local 2012-01-04 00:22:49

Using etap version "0.3.4"

1..38
ok 1 - test/cases/array
ok 2 - test/cases/array_close
ok 3 - test/cases/array_open
ok 4 - test/cases/bogus_char
ok 5 - test/cases/codepoints_from_unicode_org
ok 6 - test/cases/deep_arrays
ok 7 - test/cases/difficult_json_c_test_case
ok 8 - test/cases/doubles
ok 9 - test/cases/empty_array
ok 10 - test/cases/empty_string
ok 11 - test/cases/escaped_bulgarian
ok 12 - test/cases/escaped_foobar
ok 13 - test/cases/false
ok 14 - test/cases/false_then_garbage
ok 15 - test/cases/four_byte_utf8
ok 16 - test/cases/integers
ok 17 - test/cases/invalid_utf8
ok 18 - test/cases/isolated_surrogate_marker
ok 19 - test/cases/leading_zero_in_number
ok 20 - test/cases/lonely_minus_sign
ok 21 - test/cases/lonely_number
ok 22 - test/cases/map_close
ok 23 - test/cases/map_open
ok 24 - test/cases/missing_integer_after_decimal_point
ok 25 - test/cases/missing_integer_after_exponent
ok 26 - test/cases/non_utf8_char_in_string
ok 27 - test/cases/null
ok 28 - test/cases/null_then_garbage
ok 29 - test/cases/nulls_and_bools
ok 30 - test/cases/simple
ok 31 - test/cases/string_invalid_escape
ok 32 - test/cases/string_invalid_hex_char
ok 33 - test/cases/string_with_escapes
ok 34 - test/cases/string_with_invalid_newline
ok 35 - test/cases/three_byte_utf8
ok 36 - test/cases/true
ok 37 - test/cases/true_then_garbage
ok 38 - test/cases/unescaped_bulgarian
ok
test/002-literals.t ....

Current time local 2012-01-04 00:22:49

Using etap version "0.3.4"

1..6
ok 1 - DEC: true -> true
ok 2 - ENC: true -> true
ok 3 - DEC: false -> false
ok 4 - ENC: false -> false
ok 5 - DEC: null -> null
ok 6 - ENC: null -> null
ok
test/003-numbers.t .....

Current time local 2012-01-04 00:22:49

Using etap version "0.3.4"

1..59
ok 1 - Decoded <<"0">>.
ok 2 - Encoded 0
ok 3 - Decoded <<"-0">>.
ok 4 - Encoded 0
ok 5 - Decoded <<"1">>.
ok 6 - Encoded 1
ok 7 - Decoded <<"12">>.
ok 8 - Encoded 12
ok 9 - Decoded <<"-3">>.
ok 10 - Encoded -3
ok 11 - Decoded <<"1234567890123456789012345">>.
ok 12 - Encoded 1234567890123456789012345
ok 13 - Decoded <<"1310050760199">>.
ok 14 - Encoded 1310050760199
ok 15 - Decoded <<"1234567890123456789012345.0">>.
ok 16 - Encoded 1.2345678901234568e24
ok 17 - Decoded <<"1234567890123456789012345.0E3">>.
ok 18 - Encoded 1.2345678901234569e27
not ok 19 - Decoded <<"1234567890123456789012345E2">>.
---
description: "Decoded <<"1234567890123456789012345E2">>."
found: 1.2345678901234568e26
wanted: 123456789012345678901234500
...
ok 20 - Encoded 123456789012345678901234500
ok 21 - Decoded <<"0.000000000000000000000000000000000001">>.
ok 22 - Encoded 1.0e-36
ok 23 - Decoded <<"1.0">>.
ok 24 - Encoded 1.0
ok 25 - Decoded <<"0.75">>.
ok 26 - Encoded 0.75
ok 27 - Decoded <<"2.0123456789">>.
ok 28 - Encoded 2.0123456789
ok 29 - Decoded <<"2.4234324E24">>.
ok 30 - Encoded 2.4234324e24
ok 31 - Decoded <<"-3.1416">>.
ok 32 - Encoded -3.1416
ok 33 - Decoded <<"1E4">>.
ok 34 - Encoded 1.0e4
ok 35 - Decoded <<"1.0E+01">>.
ok 36 - Encoded 10.0
ok 37 - Decoded <<"1e1">>.
ok 38 - Encoded 10.0
ok 39 - Decoded <<"3.0E2">>.
ok 40 - Encoded 300.0
ok 41 - Decoded <<"0E3">>.
ok 42 - Encoded 0.0
ok 43 - Decoded <<"1.5E3">>.
ok 44 - Encoded 1.5e3
ok 45 - Decoded <<"2.5E-1">>.
ok 46 - Encoded 0.25
ok 47 - Decoded <<"-0.325E+2">>.
ok 48 - Encoded -32.5
ok 49 - Decoding <<"02">> returns an error.
ok 50 - Decoding <<"-01">> returns an error.
ok 51 - Decoding <<"+12">> returns an error.
ok 52 - Decoding <<"-">> returns an error.
ok 53 - Decoding <<"1.">> returns an error.
ok 54 - Decoding <<".1">> returns an error.
ok 55 - Decoding <<"1.-1">> returns an error.
ok 56 - Decoding <<"1E">> returns an error.
ok 57 - Decoding <<"1-E2">> returns an error.
ok 58 - Decoding <<"2E +3">> returns an error.
ok 59 - Decoding <<"1EA">> returns an error.
Failed 1/59 subtests
test/004-strings.t .....

Current time local 2012-01-04 00:22:49

Using etap version "0.3.4"

1..83
ok 1 - Decoded <<"""">>.
ok 2 - Encoded <<>>
ok 3 - Decoded <<""0"">>.
ok 4 - Encoded <<"0">>
ok 5 - Decoded <<""foo"">>.
ok 6 - Encoded <<"foo">>
ok 7 - Decoded <<""\"foobar\""">>.
ok 8 - Encoded <<""foobar"">>
ok 9 - Decoded <<""\n\n\n"">>.
ok 10 - Encoded <<"\n\n\n">>
ok 11 - Decoded <<""\" \b\f\r\n\t\""">>.
ok 12 - Encoded <<"" \b\f\r\n\t"">>
ok 13 - Decoded <<""foo\u0005bar"">>.
ok 14 - Encoded <<102,111,111,5,98,97,114>>
ok 15 - Decoded <<""\uD834\uDD1E"">>.
ok 16 - Encoded <<240,157,132,158>>
ok 17 - Decoded <<""\u8CA8\u5481\u3002\u0091\u0091"">>.
ok 18 - Encoded <<232,178,168,229,146,129,227,128,130,194,145,194,145>>
ok 19 - Decoding <<""foo">> returns an error.
ok 20 - Decoding <<34,0,34>> returns an error.
ok 21 - Decoding <<""\g"">> returns an error.
ok 22 - Decoding <<""\uFFFF"">> returns an error.
ok 23 - Decoding <<""\uFFFE"">> returns an error.
ok 24 - Decoding <<""\uD834foo\uDD1E"">> returns an error.
ok 25 - Decoding <<""NEIMEN F?RR FAEN!"">> returns an error.
ok 26 - Invalid utf-8: <<194,129,128>>
ok 27 - Invalid utf-8: <<34,194,129,128,34>>
ok 28 - Invalid utf-8: <<102,111,111,128,98,97,114>>
ok 29 - Invalid utf-8: <<34,102,111,111,128,98,97,114,34>>
ok 30 - Invalid utf-8: <<"?">>
ok 31 - Invalid utf-8: <<""?"">>
ok 32 - Invalid utf-8: <<237,160,129>>
ok 33 - Invalid utf-8: <<34,237,160,129,34>>
ok 34 - Invalid utf-8: <<"?">>
ok 35 - Invalid utf-8: <<""?"">>
ok 36 - Invalid utf-8: <<"?">>
ok 37 - Invalid utf-8: <<""?"">>
ok 38 - Invalid utf-8: <<224,128>>
ok 39 - Invalid utf-8: <<34,224,128,34>>
ok 40 - Invalid utf-8: <<"?">>
ok 41 - Invalid utf-8: <<""?"">>
ok 42 - Invalid utf-8: <<240,128>>
ok 43 - Invalid utf-8: <<34,240,128,34>>
ok 44 - Invalid utf-8: <<240,128,128>>
ok 45 - Invalid utf-8: <<34,240,128,128,34>>
ok 46 - Invalid utf-8: <<"?">>
ok 47 - Invalid utf-8: <<""?"">>
ok 48 - Invalid utf-8: <<248,128>>
ok 49 - Invalid utf-8: <<34,248,128,34>>
ok 50 - Invalid utf-8: <<248,128,128>>
ok 51 - Invalid utf-8: <<34,248,128,128,34>>
ok 52 - Invalid utf-8: <<248,128,128,128>>
ok 53 - Invalid utf-8: <<34,248,128,128,128,34>>
ok 54 - Invalid utf-8: <<"?">>
ok 55 - Invalid utf-8: <<""?"">>
ok 56 - Invalid utf-8: <<252,128>>
ok 57 - Invalid utf-8: <<34,252,128,34>>
ok 58 - Invalid utf-8: <<252,128,128>>
ok 59 - Invalid utf-8: <<34,252,128,128,34>>
ok 60 - Invalid utf-8: <<252,128,128,128>>
ok 61 - Invalid utf-8: <<34,252,128,128,128,34>>
ok 62 - Invalid utf-8: <<252,128,128,128,128>>
ok 63 - Invalid utf-8: <<34,252,128,128,128,128,34>>
ok 64 - Invalid utf-8: <<192,128>>
ok 65 - Invalid utf-8: <<34,192,128,34>>
ok 66 - Invalid utf-8: <<193,128>>
ok 67 - Invalid utf-8: <<34,193,128,34>>
ok 68 - Invalid utf-8: <<224,128,128>>
ok 69 - Invalid utf-8: <<34,224,128,128,34>>
ok 70 - Invalid utf-8: <<224,144,128>>
ok 71 - Invalid utf-8: <<34,224,144,128,34>>
ok 72 - Invalid utf-8: <<240,128,128,128>>
ok 73 - Invalid utf-8: <<34,240,128,128,128,34>>
ok 74 - Invalid utf-8: <<240,136,128,128>>
ok 75 - Invalid utf-8: <<34,240,136,128,128,34>>
ok 76 - Invalid utf-8: <<248,128,128,128,128>>
ok 77 - Invalid utf-8: <<34,248,128,128,128,128,34>>
ok 78 - Invalid utf-8: <<248,132,128,128,128>>
ok 79 - Invalid utf-8: <<34,248,132,128,128,128,34>>
ok 80 - Invalid utf-8: <<252,128,128,128,128,128>>
ok 81 - Invalid utf-8: <<34,252,128,128,128,128,128,34>>
ok 82 - Invalid utf-8: <<252,130,128,128,128,128>>
ok 83 - Invalid utf-8: <<34,252,130,128,128,128,128,34>>
ok
test/005-arrays.t ......

Current time local 2012-01-04 00:22:49

Using etap version "0.3.4"

1..18
ok 1 - Decoded <<"[]">>.
ok 2 - Encoded []
ok 3 - Decoded <<"[\t[\n]\r]">>.
ok 4 - Encoded [[]]
ok 5 - Decoded <<"[\t123, \r true\n]">>.
ok 6 - Encoded [123,true]
ok 7 - Decoded <<"[1,"foo"]">>.
ok 8 - Encoded [1,<<"foo">>]
ok 9 - Decoded <<"[11993444355.0,1]">>.
ok 10 - Encoded [11993444355.0,1]
ok 11 - Decoded <<"["\u00A1","\u00FC"]">>.
ok 12 - Encoded [<<"¡">>,<<"ü">>]
ok 13 - Decoding <<"[">> returns an error.
ok 14 - Decoding <<"]">> returns an error.
ok 15 - Decoding <<"[,]">> returns an error.
ok 16 - Decoding <<"[123">> returns an error.
ok 17 - Decoding <<"[123,]">> returns an error.
ok 18 - Decoding <<"[32 true]">> returns an error.
ok
test/006-maps.t ........

Current time local 2012-01-04 00:22:50

Using etap version "0.3.4"

1..15
ok 1 - Decoded <<"{}">>.
ok 2 - Encoded {[]}
ok 3 - Decoded <<"{"foo": "bar"}">>.
ok 4 - Encoded {[{<<"foo">>,<<"bar">>}]}
ok 5 - Decoded <<"\n\n{"foo":\r "bar",\n "baz"\t: 123 }">>.
ok 6 - Encoded {[{<<"foo">>,<<"bar">>},{<<"baz">>,123}]}
ok 7 - Decoding <<"{">> returns an error.
ok 8 - Decoding <<"{,}">> returns an error.
ok 9 - Decoding <<"{123:true}">> returns an error.
ok 10 - Decoding <<"{false:123}">> returns an error.
ok 11 - Decoding <<"{:"stuff"}">> returns an error.
ok 12 - Decoding <<"{"key":}">> returns an error.
ok 13 - Decoding <<"{"key": 123">> returns an error.
ok 14 - Decoding <<"{"key": 123 true">> returns an error.
ok 15 - Decoding <<"{"key": 123,}">> returns an error.
ok
test/007-compound.t ....

Current time local 2012-01-04 00:22:50

Using etap version "0.3.4"

1..12
ok 1 - Decoded <<"[{}]">>.
ok 2 - Encoded [{[]}]
ok 3 - Decoded <<"{"foo":[123]}">>.
ok 4 - Encoded {[{<<"foo">>,"{"}]}
ok 5 - Decoded <<"{"foo":{"bar":true}}">>.
ok 6 - Encoded {[{<<"foo">>,{[{<<"bar">>,true}]}}]}
ok 7 - Decoded <<"{"foo":[],"bar":{"baz":true},"alice":"bob"}">>.
ok 8 - Encoded {[{<<"foo">>,[]},
{<<"bar">>,{[{<<"baz">>,true}]}},
{<<"alice">>,<<"bob">>}]}
ok 9 - Decoded <<"[-123,"foo",{"bar":[]},null]">>.
ok 10 - Encoded [-123,<<"foo">>,{[{<<"bar">>,[]}]},null]
ok 11 - Decoding <<"[{}">> returns an error.
ok 12 - Decoding <<"}]">> returns an error.
ok
test/008-halfword.t ....

Current time local 2012-01-04 00:22:50

Using etap version "0.3.4"

ok 1 - 1 =:= 1
ok 2 - 1 == 1
1..2
ok

Test Summary Report

test/003-numbers.t (Wstat: 0 Tests: 59 Failed: 1)
Failed test: 19
Files=8, Tests=233, 2 wallclock secs ( 0.11 usr 0.02 sys + 1.54 cusr 0.23 csys = 1.90 CPU)
Result: FAIL
make: *** [etap] Error 1

Hello,

When using 'git' as application version in jiffy.app.src, rebar gets really confused when jiffy is used as a dependency. The proper fix would be to fix rebar, but a workaround could be to use a real version string as the version in jiffy.app.src.

Thanks
Knut

Would it be possible to include a function for validating EJSON and maybe also JSON Strings?

The following sequence fails :

jiffy:encode(jiffy:decode(<<"\"\\u8ca8\\u5481\\u3002\\u0091\\u0091\"">>), [uescape]).

The bug is in encoder.c, line 245. ulen is wrongly set to the result of utf8_esc_len. Instead, it should be left alone (ulen is used to iterate on the source string) and esc_extra should be incremented with the result of utf8_esc_len function.

Hi,

I have some data with integers bigger than 64 bits. Jiffy decodes these fine, but encoding them back again behaves a bit unexpected. I get an iolist instead of the binary I've come to expect.

It seems what happens is that in the main encode function, the type doesn't match and it then calls enc_unknown. Maybe this is expected behaviour. Is it something I can rely on?

jiffy:encode({[{key, 11111111111111111111}]}).
[<<"{\"key\":">>,<<"11111111111111111111">>,<<"}">>]

What's the license Jiffy is licensed under?

Getting this on gcc 4.8.1:

Compiling c_src/double-conversion/fast-dtoa.cc
c_src/double-conversion/fast-dtoa.cc: In function ‘bool double_conversion::FastDtoa(double, double_conversion::FastDtoaMode, int, double_conversion::Vector<char>, int*, int*)’:
c_src/double-conversion/fast-dtoa.cc:254:60: error: array subscript is below array bounds [-Werror=array-bounds]
   while (number < kSmallPowersOfTen[exponent_plus_one_guess]) {
                                                            ^
c_src/double-conversion/fast-dtoa.cc:254:60: error: array subscript is below array bounds [-Werror=array-bounds]
   while (number < kSmallPowersOfTen[exponent_plus_one_guess]) {
                                                            ^
cc1plus: all warnings being treated as errors

There have been a number of reports of false positives for this particular warning on older versions of gcc, so this may fall into that category. If it's not obvious to the author(s) why this is firing, I can look deeper.

http://permalink.gmane.org/gmane.comp.lang.erlang.general/54419
I'm suffering from the same problem.

I tried to build Jiffy with rebar compile, but it gave me an error saying /Werror not recognized. I removed the flag from the rebar.config file, and everything worked.

I was using Windows 7 64 bit with Microsoft Visual Studio 2010 Express. Please let me know if you need any additional info reproducing or testing this.

11> jiffy:decode(<<"{"foo": "bar"}">>).

=ERROR REPORT==== 23-Mar-2013::23:05:39 ===
The on_load function for module jiffy returned {error,
{load_failed,
"Failed to load NIF library: 'dlopen(deps/jiffy/priv/jiffy.so, 2): no suitable image found. Did find:\n\tdeps/jiffy/priv/jiffy.so: mach-o, but wrong architecture'"}}
** exception error: undefined function jiffy:decode/1

Just a thought: maybe it would be a good idea to put some constraints on the input size (which should be overridable, of course)? The reason is that a long-running NIF will ruin an Erlang VM's scheduler, so it's not too wise to throw too much JSON in a single call. We can prevent this mistake by uploading an encoding/decoding to a separate thread or by putting constraints. I think the latter is easier.

I have a binary data. I can use mochijson (https://github.com/mochi/mochiweb/blob/master/src/mochijson.erl) to decode it, but failed if I use jiffy.

This is my code.

1> Bin = <<91,34,226,128,140,226,128,141,226,128,142,226,128,143,226,128,168,226,128,169,226,128,170,226,128,171,226,128,172,226,128,173,226,128,174,226,128,175,226,129,160,226,129,161,226,129,162,226,129,163,226,129,164,226,129,165,226,129,166,226,129,167,226,129,168,226,129,169,226,129,170,226,129,171,226,129,172,226,129,173,226,129,174,226,129,175,239,191,176,239,191,177,239,191,178,239,191,179,239,191,180,239,191,181,239,191,182,239,191,183,239,191,184,239,191,185,239,191,186,239,191,187,239,191,188,239,191,189,239,191,190,239,191,191,34,93>>.

2> jiffy:decode(Bin).
** exception throw: {error,{129,invalid_string}}
     in function  jiffy:decode/1 (src/jiffy.erl, line 13)

3> mochijson:decode(Bin).
{array,[[8204,8205,8206,8207,8232,8233,8234,8235,8236,8237,
         8238,8239,8288,8289,8290,8291,8292,8293,8294,8295,8296,8297,
         8298,8299,8300,8301|...]]}

What do you think? afaik the usage of null atom in Erlang code is much less common.

03f2b30 breaks compilation in CentOS 5.

==> jiffy (compile)
Compiling c_src/decoder.c
cc1: warnings being treated as errors
c_src/decoder.c: In function ‘decode_iter’:
c_src/decoder.c:736: warning: dereferencing type-punned pointer will break strict-aliasing rules

Hi!

Parsing {"a":"b"} followed by 2040 or more space characters leads to a VM crash. The following snippet illustrates it:

$ erl -pa jiffy/ebin     
Erlang R15B01 (erts-5.9.1) [source] [64-bit] [async-threads:0] [kernel-poll:false]

Eshell V5.9.1  (abort with ^G)
1> jiffy:decode(<<"{\"a\":\"b\"}                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        ">>).
zsh: segmentation fault (core dumped)  erl -pa jiffy/ebin

The code was executed on a x86-64 Linux system. Please let me know if you need any more information.

Best,
Michał

Trying to compile jiffy on Debian squeeze I get the following errors:

Compiling c_src/decoder.c
c_src/decoder.c: In function ‘dec_number’:
c_src/decoder.c:552: warning: implicit declaration of function ‘enif_make_int64’
Compiling c_src/encoder.c
c_src/encoder.c:365: error: expected declaration specifiers or ‘...’ before ‘ErlNifSInt64’
c_src/encoder.c: In function ‘enc_long’:
c_src/encoder.c:374: error: ‘val’ undeclared (first use in this function)
c_src/encoder.c:374: error: (Each undeclared identifier is reported only once
c_src/encoder.c:374: error: for each function it appears in.)
c_src/encoder.c: In function ‘encode’:
c_src/encoder.c:519: error: ‘ErlNifSInt64’ undeclared (first use in this function)
c_src/encoder.c:519: error: expected ‘;’ before ‘lval’
c_src/encoder.c:624: warning: implicit declaration of function ‘enif_get_int64’
c_src/encoder.c:624: error: ‘lval’ undeclared (first use in this function)
c_src/encoder.c:625: error: too many arguments to function ‘enc_long’
ERROR: cc -c  -g -Wall -g -Wall -fPIC  -I/usr/lib/erlang/lib/erl_interface-3.7/include -I/usr/lib/erlang/erts-5.8/include   c_src/encoder.c -o c_src/encoder.o failed with error: 1 and output:
c_src/encoder.c:365: error: expected declaration specifiers or ‘...’ before ‘ErlNifSInt64’
c_src/encoder.c: In function ‘enc_long’:
c_src/encoder.c:374: error: ‘val’ undeclared (first use in this function)
c_src/encoder.c:374: error: (Each undeclared identifier is reported only once
c_src/encoder.c:374: error: for each function it appears in.)
c_src/encoder.c: In function ‘encode’:
c_src/encoder.c:519: error: ‘ErlNifSInt64’ undeclared (first use in this function)
c_src/encoder.c:519: error: expected ‘;’ before ‘lval’
c_src/encoder.c:624: warning: implicit declaration of function ‘enif_get_int64’
c_src/encoder.c:624: error: ‘lval’ undeclared (first use in this function)
c_src/encoder.c:625: error: too many arguments to function ‘enc_long’

Erlang version is R14A (erts-5.8), gcc is at 4.4.5.

Hi! i trying to generate this strange json, where data - is json string instide json object.

{"event":"hello","data":"{"presence":{"count":1,"ids":["462127"],"hash":{"462127":{"first":"first_name","last":"last_name"}}}}","channel":"channel"}

Any ideas how to generate it with jiffy?

When including the project, and downloading/compiling with rebar, the project outputs a file jiffy_drv.so to the /priv dir. However jiffy:decode() reports

1> application:start(jiffy).
ok
2>
=PROGRESS REPORT==== 9-Feb-2013::11:15:57 ===
application: jiffy
started_at: nonode@nohost
2> jiffy:decode(<<"[]">>).

=ERROR REPORT==== 9-Feb-2013::11:16:16 ===
The on_load function for module jiffy returned {error,
{load_failed,
"Failed to load NIF library: 'dlopen(deps/jiffy/priv/jiffy.so, 2): image not found'"}}
** exception error: undefined function jiffy:decode/1
3>

Renaming the .so file solves the issue..

I clone jiffy into my workspace, compile it without change the rebar.config file, and I also add "D:\VS6\VC98\Bin" to my path enviroment, but there is an error:
==> jiffy (compile)
INFO: Skipped src/jiffy_utf8.erl
INFO: Skipped src/jiffy.erl
INFO: sh info:
cwd: "E:/Workspace/jiffy"
cmd: cl.exe /c /Wall /DWIN32 /D_WINDOWS /D_WIN32 /DWINDOWS /Zi /Wall -I
e:/ERL5931.1/lib/erl_interface-3.7.9/include -Ie:/ERL5931.1/erts-5.9.3.1/inclu
de c_src/decoder.c /Foc_src/decoder.o
Compiling e:/Workspace/jiffy/c_src/decoder.c
ERROR: compile failed while processing E:/Workspace/jiffy: rebar_abort

how can I build the jiffy.dll in windows?

We're currently using jiffy in our application and we see some test regressions when using the half word emulator.

Specifically we're observing that some function clauses stop matching in other sections of our code, if we exclude jiffy calls from our code by providing directly the parsed structures all our test go back to succeeding, this behavior is observed only using the half word emulator and everything works just fine using the pure 32 or 64 bits VMS.

We're unable to provide a breaking test that can be published but we can provide access to an environment where is possibile to observe the behavior.

Issuing jiffy:decode("\""). crashes the VM.

Setup tested:

• on master;
• R15B03, R16B;
• Mac OS X, Linux.

$ erl -pa ebin/
Erlang R15B03 (erts-5.9.3.1) [source] [64-bit] [smp:2:2] [async-threads:0] [hipe] [kernel-poll:false]

Eshell V5.9.3.1  (abort with ^G)
1> jiffy:decode("\"").
Bus error: 10

$ erl -pa ebin/
Erlang R16B (erts-5.10.1) [source] [smp:2:2] [async-threads:10] [hipe] [kernel-poll:false]

Eshell V5.10.1  (abort with ^G)
1> jiffy:decode("\"").
Segmentation fault: 11

Hi,

We have noticed a problem when running on a 32 bit machine. It appears that big numbers are somehow truncated to 27 bits.

Test on a 64 bit machine:
Erlang R14B03 (erts-5.8.4) [source] [64-bit halfword] [smp:8:8] [rq:8] [async-threads:0] [kernel-poll:false]
jiffy:encode({[{foo, 1310050760199}]})
{ok,<<"{"foo":1310050760199}">>}

On a 32 bit machine:
Erlang R14B01 (erts-5.8.2) [source] [rq:1] [async-threads:0] [kernel-poll:false]
jiffy:encode({[{foo, 1310050760199}]}).
{ok,<<"{"foo":85734919}">>}

Thanks
Knut

The version pattern matching in rebar.config is invalid with latest versions of Erlang.

Will provide a PR asap but wanted to report it to not forget ;)

Is there a particular reason why jiffy expects and produces these extra curly braces which make it super annoying to work with? In several projects I have now an unpack and pack function which will remove those prior to encoding and after decoding so I get regular proplist representation.

I'm quite new to erlang, so chances are great that I missed an essential part. I can't imagine that you enjoy those extra curly braces around everything which is why I hope that there is something to make working with those structures more pleasant.

jiffy:encode({[{"one",1},{"two",2},{"three",3}]}).
** exception throw: {error,invalid_object_member_key}
in function jiffy:encode/2 (src/jiffy.erl, line 34)

jiffy:encode({[{<<"one">>,1},{<<"two">>,2},{<<"three">>,3}]}).
<<"{"one":1,"two":2,"three":3}">>

Something goes wrong when decoding lists of integers :

jiffy:decode(<<"[ 42, 23 ]">>). gives [42,23] which is ok, AFAIC

jiffy:decode(<<"[ 42, 43 ]">>). gives "*+" which is not what I was expecting, as [42, 43] is legal JSON code, describing the list of integers 42 and 43.

Note that jiffy:decode("[ 42, true, 43 ]"). gives [42,true,43] which is ok.

Am I missing something ?

I should think about counting function calls inside the nif and then bump the Erlang process's reduction count with erlang:bump_reductions/1 based on that value.

I got function not found when I call jiffy:encode/1 and jiffy:decode/1, anyone one know whats problem?

1> jiffy:encode(<<>>).
** exception error: undefined function jiffy:encode/1
2>
=ERROR REPORT==== 13-Aug-2014::13:47:04 ===
The on_load function for module jiffy returned {error,
{bad_lib,
"Function not found jiffy:nif_decode/1"}}

jiffy:encode(<<"/test/essai">>) return "/test/essai" . Maybe an issue ? I didn't find a way to unescape it...

Hi I get the following error when trying to compile on windows with a MinGW setup 32 bits

==> jiffy (compile)
Compiling c_src/decoder.c
cc: error: /Wall: No such file or directory
cc: error: /DWIN32: No such file or directory
cc: error: /D_WINDOWS: No such file or directory
cc: error: /D_WIN32: No such file or directory
cc: error: /DWINDOWS: No such file or directory
ERROR: cmd /q /c cc -c /Wall /DWIN32 /D_WINDOWS /D_WIN32 /DWINDOWS -g -Wall -fPIC -Ic:/PROGRA2/ERL5101.1/lib/erl_interface-3.7.11/include -Ic:/PROGRA2/ERL5101.1/erts-5.10.1/include c_src/decoder.c -o c_src/decoder.o failed with error: 1 and output:
cc: error: /Wall: No such file or directory
cc: error: /DWIN32: No such file or directory
cc: error: /D_WINDOWS: No such file or directory
cc: error: /D_WIN32: No such file or directory
cc: error: /DWINDOWS: No such file or directory

It seems that jiffy isn't correctly encoding numbers:

1> N = jiffy:decode(<<"1.0">>).
1.0
2> jiffy:encode(N).
<<"1">>
3> N1 = jiffy:decode(<<"1.1">>).
1.1
4> jiffy:encode(N1).
<<"1.1000000000000000888">>

sound like there is a problem somewhere in the precision.

I'm getting an error when I try to compile on Windows 7 - I have rebar installed and on my path, along with cl from Visual Studio 2010 (which I understand should be recent enough). However it seems that the file "assert.h" is missing:

$ rebar compile
==> jiffy (compile)
Compiling c_src/decoder.c
Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 16.00.30319.01 for 80x86
Copyright (C) Microsoft Corporation. All rights reserved.

decoder.c
c_src/decoder.c(4) : fatal error C1083: Cannot open include file: 'assert.h': No such file or directory
ERROR: compile failed while processing c:/Users/gbhowarto/code/orca/deps/jiffy: rebar_abort

IN prod we don't really need proper. Would be cool if you could make it an optional dependency.

I get a segfault 11 when using jiffy on some database in refuge :

gdb) bt
#0  0x00007fff8a592df2 in select$DARWIN_EXTSN ()
#1  0x00000000111c1551 in erts_sys_main_thread () at sys.c:2905
#2  0x00000000110de104 in erl_start (argc=26, argv=0x7fff60cc3308) at erl_init.c:1483
#3  0x6e65622f73726573 in ?? ()
(gdb) c
Continuing.
Reading symbols for shared libraries . done

Program received signal EXC_BAD_ACCESS, Could not access memory.
Reason: 13 at address: 0x0000000000000000
[Switching to process 1049 thread 0x2803]
0x00000000110d8ad3 in bf_unlink_free_block (allctr=0x7f8e19864fc0, block=0x13031050, flags=0) at erl_bestfit_alloc.c:792
792     LIST_NEXT(LIST_PREV(x)) = LIST_NEXT(x);
(gdb) c
Continuing.

Program received signal EXC_BAD_ACCESS, Could not access memory.
Reason: 13 at address: 0x0000000000000000
0x00000000110d8ad3 in bf_unlink_free_block (allctr=0x7f8e19864fc0, block=0x13031050, flags=0) at erl_bestfit_alloc.c:792
792     LIST_NEXT(LIST_PREV(x)) = LIST_NEXT(x);
(gdb) bt
#0  0x00000000110d8ad3 in bf_unlink_free_block (allctr=0x7f8e19864fc0, block=0x13031050, flags=0) at erl_bestfit_alloc.c:792
#1  0x00000000110d3002 in do_erts_alcu_realloc () at erl_alloc_util.c:3568
#2  0x0000000011a2dfbe in enc_done (e=0xb09a9cc0, value=0xb09a9c78) at c_src/encoder.c:153
#3  0x0000000011a2ec54 in encode (env=0xb09a9dd0, argc=2, argv=0x1178d1c0) at c_src/encoder.c:704
#4  0x00000000111a8469 in process_main () at beam_emu.c:3391
Previous frame inner to this frame (gdb could not unwind past this frame)

you can reproduce it using refuge master branch with the bookmark db in the data.tgz archive:

1. Get the refuge sources
2. make rel to create a release
3. Put the content of the data.tgz archibe in rel/refuge/data
4. launch refuge ./rel/refuge/bin/refuge console

Then try ot access to the bookmark db:

curl http://0.0.0.0:5984/bookmarks/_design/bookmarks

It will end with a segfaul. Using the ejson branch of refuge that uses the ejson encoder fix the issue.

It seems that actually jiffy could only decode binaries. It would be useful if it could accept any iolists instead. For example there is a lot of case to handle in couchdb due to this :

https://github.com/refuge/couch_replicator/tree/jiffy
https://github.com/refuge/couch_mrview/tree/jiffy
https://github.com/refuge/couch/tree/jiffy

Erlang R15B03 (erts-5.9.3) [source] [smp:4:4] [async-threads:0] [hipe] [kernel-poll:false]

Eshell V5.9.3  (abort with ^G)
1> l(jiffy).
{error,on_load_failure}

=ERROR REPORT==== 17-Apr-2014::11:07:54 ===
The on_load function for module jiffy returned {error,
                                                {load_failed,
                                                 "Failed to load NIF library: 'dlopen(./priv/jiffy.so, 2): no suitable image found.  Did find:\n\t./priv/jiffy.so: mach-o, but wrong architecture'"}}
2>

$ uname -a
Darwin coolchevy.favorit 13.1.0 Darwin Kernel Version 13.1.0: Thu Jan 16 19:40:37 PST 2014; root:xnu-2422.90.20~2/RELEASE_X86_64 x86_64

May be I need specify some compile args in rebar.config?
Thanks

I get a compile error with gcc 4.8

==> jiffy (compile)
Compiling c_src/doubles.cc
In file included from c_src/double-conversion/double-conversion.h:31:0,
                 from c_src/doubles.cc:1:
c_src/double-conversion/utils.h: In function ‘Dest double_conversion::BitCast(const Source&)’:
c_src/double-conversion/utils.h:299:16: error: typedef ‘VerifySizesAreEqual’ locally defined but not used [-Werror=unused-local-typedefs]
   typedef char VerifySizesAreEqual[sizeof(Dest) == sizeof(Source) ? 1 : -1];
                ^
cc1plus: all warnings being treated as errors
ERROR: compile failed while processing /home/tburdick/wti/inkgraph/deps/jiffy: rebar_abort
make: *** [build] Error 1