michalmuskala / jason Goto Github PK
View Code? Open in Web Editor NEWA blazing fast JSON parser and generator in pure Elixir.
License: Other
A blazing fast JSON parser and generator in pure Elixir.
License: Other
I think we should provide HiPE support. I was thinking something like this in the modules where it would make sense:
if System.get_env("JASON_HIPE") do
@compile :native
@compile {:hipe, [:verbose, :o3]}
end
Thoughts? This may perhaps become less relevant if #9 happens.
We should be able to send a flag to the decoder that will preserve atoms. So when they are written they will be written with the :
in front of them and when they are read, if there is a leading : it should be read back as an atom. The user should be able to use only existing atoms as well and the code should throw if the atom doesn't exist. I would recommend something like an option atoms with the possible values of :none (the default now), :create
which will call string_to_atom
on decode and :existing
which will call string_to_existing_atom
on decode. The encoder should also be able to take such an option.
Due to some unexpected errors, I got a lot of rollbax crashes and therefore no information about my errors. They weren't sent because of the following:
09:40:56.159 [error] GenServer Rollbax.Client terminating
** (Protocol.UndefinedError) protocol Jason.Encoder not implemented for {"id", "2"}, Jason.Encoder protocol must always be explicitly implemented. This protocol is implemented for: Any, Atom, BitString, Date, DateTime, Decimal, Float, Integer, Jason.Fragment, List, Map, NaiveDateTime, Time
lib/encoder.ex:128: Jason.Encoder.Any.encode/2
lib/encode.ex:138: Jason.Encode.list/3
lib/encode.ex:172: Jason.Encode.map_naive_loop/3
lib/encode.ex:173: Jason.Encode.map_naive_loop/3
lib/encode.ex:162: Jason.Encode.map_naive/3
lib/encode.ex:172: Jason.Encode.map_naive_loop/3
lib/encode.ex:173: Jason.Encode.map_naive_loop/3
lib/encode.ex:162: Jason.Encode.map_naive/3
Is there an easy way of implementing the encoder for other (standard) types?
Are there any plans for having streaming support?
http://seriot.ch/parsing_json.php
Was reading this on hacker news and thought it was a good set of extra tests that might be useful.
Discussion: https://news.ycombinator.com/item?id=16897061
I find that I'm filtering maps before marshalling data in the encoder. This requires me to do an Map.into(%{})
before sending it to |> Jason.Encode.map
Do you think it would be worth doing a Jason.Encode.keyword
?
would be happy to PR it myself.
I'm trying to parse this json with Jason, and I'm getting:
{:error,
%Jason.DecodeError{
data: "<html xmlns....<> ...,
position: 0,
token: nil
}}
Appreciate the help trying to figure this one out. It's the first time Jason doesn't parse something for me.
With Poison, I'm able to define a custom encoder for Date
by doing the following:
defimpl Poison.Encoder, for: Date do
def encode(date, options) do
Blah.CustomDateFormatter.render_date(date)
|> Poison.Encoder.encode(options)
end
end
I have a custom encoder because Date.to_iso8601
doesn't support years > 9999, so we need to catch this case and encode the Date in a different manner.
When I try to do the same thing with Jason, it doesn't seem to work. It seems this is because Jason.Encode
already defines what to do with Date
(also see this comment).
defimpl Jason.Encoder, for: Date do
def encode(value, _opts) do
"foo!"
end
end
Is there any way to specify a custom encoding for Date
?
After noticing how fast Jason was compared to Poison, I'd love to use it! This is the only thing holding us back, unfortunately.
I understand the reasons you've not implemented this as listed at:
The international standard for medical messages is currently moving to FHIR: http://hl7.org/fhir/
They’ve made certain… poor choices (like not including the version id in any message documents).
One of these is that if a message is delivered in JSON format, floats must be encoded as numbers in the JSON with precision preserved.
I know this breaks generic encode/decode round-trip systems, but alas, it's the standard we have to adhere to for medicine. And yes, it's really stupid :(
Do you think I should make a 'healthcare' fork of your library instead of trying to get the option added?
It's pretty annoying that I always need to preload associations for structs before encoding.
Is there some way to just skip certain field if it was not loaded?
Related error says:
Or choose to not encode the association when converting the struct to JSON by explicitly listing the JSON fields in your schema
It's definitely not solution for me because I need sometimes either struct with associations and without to encode
why phoenix choose jason as json_library by default ? there is some incompatible with the old version, jason can’t encode my ecto schema (while older version can)
Hello, we're replacing Poison by Jason in a project and it all works great but we found a test case where Poison didn't complain, but Jason does, which involves new lines inside JSON strings.
I've added a test to reproduce it here:
Now, for a little more context, we use helpers like that in out tests:
defp unauthorized_response_body do
"""
{
"error": "invalid_client",
"error_description": "Client authentication failed due to unknown client, no client
authentication included, or unsupported authentication method."
}
"""
end
The newline there is just for readability purposes, but it's what raises the decoding error.
I think that JSON allows to include \n
in strings, and Poison allows that:
iex(4)> json = """
...(4)> {
...(4)> "error": "invalid_client",
...(4)> "error_description": "Client authentication failed due to unknown client, no client
...(4)> authentication included, or unsupported authentication method."
...(4)> }
...(4)> """
"{\n \"error\": \"invalid_client\",\n \"error_description\": \"Client authentication failed due to unknown client, no client\n authentication included, or unsupported authentication method.\"\n}\n"
iex(5)> Poison.decode(json)
{:ok,
%{"error" => "invalid_client",
"error_description" => "Client authentication failed due to unknown client, no client\n authentication included, or unsupported authentication method."}}
Decimal
should result in unquoted values in the produced JSON. Any Decimal type should be aligned with the behaviour of Integer and Float.
Currently we have the following:
Jason.encode! %{decimal: Decimal.new("1.23")}
> {"decimal":"1.23"}
Can we have Jason not put quotes around the value and produce the following JSON instead?
{"decimal":1.23}
Given that Decimal is in fact a number, and none of the ECMA/RFC standards define number (decimal/integer) boundaries, it is strange to map it to a string.
We recently moved from Poison
to Jason
, we changed Cloudfront signer with:
Jason.encode!(
%{
"Statement" => [
%{
"Resource" => url,
"Condition" => %{
"DateLessThan" => %{
"AWS:EpochTime" => expires
}
}
}
]
}
)
With Poison
the order is preserved, after the switch to Jason
, Resource
and Condition
are reversed, which lead to an Access denied when we try to access the resource due to an invalid signature.
Do you think an option to preserve the order could be useful ?
When encoding or decoding a struct it would be useful to add the ability to preserve the struct field so that it is not lost. Right now I have to decode then go back through and put the struct field back in and fix the data because it is lost.
When you @derive [Jason.Encoder]
and use keys in struct starting from underscore like :'_name'
it will lead to compilation warnings.
warning: the underscored variable "_name" (context Jason.Encoder.Any) is used after being set. A leading underscore indicates that the value of the variable should be ignored. If this is intended please rename the variable to remove the underscore
Now that phoenix 1.4 is going through the release process and Jason is the new default encoder, what if anything need to be changed in the documentation to support this?
For the code snippet below:
%{id: complex.id,
name: complex.name,
street: complex.street,
locality: complex.locality,
city: complex.city,
state: complex.state,
pincode: complex.pincode,
test_payload: {
test_key: "test_value"
}
Phoenix framework reports a syntax error before: test_key
The JSON rendered if I remove the sub section (test_payload
) and directly list the test_key
is valid however.
We should be at least able to define a property where Antidote.decode!(Antidote.encode!(data)) == data
for some types of data. We should also define one for just strings to test all the various string escaping options.
defmodule Foo do
@derive Jason.Encoder
defstruct [:_name]
end
this will generate a warning about using an underscored variable - we shouldn't suppress the warning or use something else for naming the variables.
Hi,
Matrix specifications defines canonical JSON as JSON where JSON encoding with dictionary keys lexicographically sorted by unicode codepoint.
Does Jason support key ordering when rendering a JSON struct ?
unexpected behaviour:
{:error, _} = Jason.encode({:invalid, :value})
throws Protocol.UndefinedError
.
I would expect that encode
returns an error tuple. Is there a reason that Jason.Encode.encode
doesn't catch this error?
Hi,
I have hard time connecting Jason with Phoenix app, is there a chance to update docs to include default Phoenix setup file names? The main issue is that I am not sure where to place each part of the code.
I am new to Elixir/Phoenix, so that is probably the reason.
Thanks.
If an application defines protocol for another application's struct, then does it introduce side-effect - impacting other applications?
# application Foo
module Foo.Hello do
require Protocol
Protocol.derive(Jason.Encoder, Plug.Upload)
end
# in the same application
module Foo.World do
end
# another application wants to use different encoding behavior
module Bar.Hello do
end
Ideally, it would be nice to specify how to convert them when calling Jason.encode/1
.
Or... at least it would be nice if the change is only applied to the application where it's made.
Hello Michał,
I was wondering if there are any plans to provide an option for selecting key formatting for maps.
Best regards,
Dennis
==> antidote
Compiling 7 files (.ex)
warning: function Decimal.to_string/1 is undefined (module Decimal is not available)
lib/encoder.ex:154
Generated antidote app
JSONL
documents, ala http://jsonlines.org, are streams of valid JSON documents, separated by newlines ("\n"
).
I know it's pretty easy to parse these in Elixir itself (e.g. StringIO.open(jsonl) |> elem(1) |> IO.binstream(:line) |> Stream.map(&Jason.decode!/1)
), but a convenience method would be nice.
As well, there may be an appreciable amount of redundant work done by the above approach, compared to having Jason manage document-splitting while parsing. Embedding the logic for handling top-level newline tokens in the document may be quite a lot faster. Something to benchmark!
To keep the speed, the pretty-printing has to be done outside of the main encoding loop. This means, adding an additional pass, after the whole encoding is done, to pretty-print the result. This will be slower, but I don't think the speed is extremely important when using prettified JSON.
Hello! This may seem an obnoxious suggestion but it is intended to be both friendly and perfectly dismissable :)
Would it be possible to convert this library to Erlang and rebar3? I ask because I believe this to be the best JSON encoding/decoding library available in the ecosystem, and Erlang/LFE/Alpaca/Gleam etc users would benefit tremendously from it.
It is currently possible to use the Elixir library but having to pull in the Elixir compiler and stdlib will deter many users.
See also: The telemetry library, which recently went through the same transition beam-telemetry/telemetry#20
Thank you :)
Context: davisp/jiffy#130 (comment)
The __struct__
element is special in Elixir for marking a map which has only atoms as keys as being a struct. The user should have the option of telling Jason to save the struct element in the JSON as a string (which can be done with tagging required fields) but also should decode the JSON key/value pair "__struct__": "MyModule"
as %{__struct__: MyModule
with a value that is an atom using string_to_atom. This would give the user the option to simply pass along the struct element and then decode instead of having to laboriously go through the JSON changing all the maps read. This will make Jason extremely useful with document oriented databases where the JSON is being saved and read frequently.
I really love the speed of Jason 😄 . However, I am having trouble with the default implementation while using it in hex packages honeybadger-io/honeybadger-elixir#160 . How can I use jason
while still being able to encode json for arbitrary structs that are passed?
This could be an optional mode for the parser.
The following example shows a case where the new pretty printer not only fails to pretty print, but also changes the data. Note the added whitespace to the URIs.
iex> Jason.encode!([%{"@id" => "http://a/b", "http://a/c" => [%{"@id" => "http://a/d"}]}], pretty: true)
[
{
"@id":"http: //a/b","http: //a/c":[{"@id":"http: //a/d"}]}]
https://github.com/michalmuskala/antidote/blob/master/lib/encode.ex#L44-L48
We should probably remove naive
escaping - it's not that big of a speed difference actually.
Jason fails to decode this JSON file: https://github.com/AndrewDryga/paasaa/blob/master/priv/scripts.json. I have verified that it's valid by ECMA-404 and all other standards supported by https://jsonformatter.curiousconcept.com/.
Hey Whenever I try to serialize a list of one element it gives me that output:
Jason.decode("[39]")
{:ok, '\''}
If the list has more than one element then i get the correct list.
Jason.decode("[39, 1]")
{:ok, [39, 1]}
And Finally if i encode it with Json and then decoded i get the same problem.
{:ok, l} = Jason.encode([39])
{:ok, "[39]"}
Jason.decode(l)
{:ok, '\''}
Does anyone have an idea how I can fix this
`
The idea would be to have a native implementation in either C or Rust. The package would be an optional dependency of jason
. The Jason.encode/decode
calls would automatically and seamlessly switch to the native code if available. This means a library author could just depend on jason
and the developer of the final application could decide if they prefer the Elixir or native implementation.
Things that should be particularly nice for performance should be leveraging SIMD in escaping and some parts of parsing (similar to what is done in rapidjson).
No reason other than pure interest.
It should be easy using Antidote.Helpers.json_map_take
macro.
Let's take this map as example:
%{"foo" => "bar", :foo => "buz"}
When encoding Jason would keep both keys producing an invalid map. Poison did the same, but with Poison ordering was different.
iex(6)> Jason.encode!(map)
"{\"foo\":\"bar\",\"foo\":\"buz\"}"
iex(7)> Poison.encode!(map)
"{\"foo\":\"buz\",\"foo\":\"bar\"}"
I think we should warn in those cases, otherwise, we going to have issues with decoding:
iex(8)> Jason.decode!(Jason.encode!(map))
%{"foo" => "bar"}
iex(9)> Poison.decode!(Poison.encode!(map))
%{"foo" => "buz"}
and some JavaScript:
# Decode Jason-encoded JSON (notice that it's different from how Jason would decode it)
JSON.parse("{\"foo\":\"bar\",\"foo\":\"buz\"}")
{foo: "buz"}
# Decode Poison-encoded JSON
JSON.parse("{\"foo\":\"buz\",\"foo\":\"bar\"}")
{foo: "bar"}
We had a bug where we generated a map with duplicate keys (by reducing it), it worked with Poison because of ordering and stopped after we replaced Poison with Jason.
I'm not sure if there is any other than emitting a warning way to deal with those through.
It looks like passing pretty: false
to the encoder raises an exception.
pry(9)> Jason.encode(%{})
{:ok, "{}"}
pry(10)> Jason.encode(%{}, pretty: true)
{:ok, "{}"}
pry(11)> Jason.encode(%{}, pretty: false)
** (FunctionClauseError) no function clause matching in Jason.Formatter.parse_opts/5
The following arguments were given to Jason.Formatter.parse_opts/5:
# 1
false
# 2
" "
# 3
"\n"
# 4
nil
# 5
" "
Attempted function clauses (showing 2 out of 2):
defp parse_opts([{option, value} | opts], indent, line, record, colon)
defp parse_opts([], indent, line, record, colon)
(jason) lib/formatter.ex:122: Jason.Formatter.parse_opts/5
(jason) lib/formatter.ex:66: Jason.Formatter.pretty_print_to_iodata/2
(jason) lib/jason.ex:212: Jason.do_encode/2
(jason) lib/jason.ex:125: Jason.encode/2
(stdlib) erl_eval.erl:677: :erl_eval.do_apply/6
(iex) lib/iex/evaluator.ex:249: IEx.Evaluator.handle_eval/5
(iex) lib/iex/evaluator.ex:229: IEx.Evaluator.do_eval/3
(iex) lib/iex/evaluator.ex:207: IEx.Evaluator.eval/3
(iex) lib/iex/evaluator.ex:94: IEx.Evaluator.loop/1
(iex) lib/iex/evaluator.ex:24: IEx.Evaluator.init/4
(iex) lib/iex/pry.ex:64: IEx.Pry.pry/2
(absinthe) lib/mix/tasks/absinthe.schema.json.ex:67: Mix.Tasks.Absinthe.Schema.Json.run/1
(mix) lib/mix/task.ex:316: Mix.Task.run_task/3
(mix) lib/mix/cli.ex:79: Mix.CLI.run_task/2
(elixir) lib/code.ex:767: Code.require_file/2
Hi all,
while the JSON spec doesn't support the handling of NaNs or +-Infinity, quite a few JSON parsers (most notably the default python parser) support these by default. Consider, for example, the following python snippet:
>>> import json
>>> data = [float('nan'), float('inf'), float('-inf')]
>>> data
[nan, inf, -inf]
>>> json.dumps(data)
'[NaN, Infinity, -Infinity]'
While this is not conforming to spec, it is something we need to deal with when integrating against services that we do not control.
Currently, I am not aware of any nice workarounds. Thus, I would like to propose a small extension to Jason to allow handling such cases. A quick look at the decoder suggests that adding handlers for 'NaN', "Infinity" and '-Infinity' would be straight forward. This could also be controlled by a decoding option whether or not to decode such json (default should be false). Internally, these could be mapped to atoms. Since this branch is only triggered in "error cases", this change would not have any impact on performance for json that follows the spec.
On encoding, as far as I see, all that would be needed would be a custom escape function.
Looking forward to feedback.
As a newer Elixir user, after some research, this library seems to be targeted as the de-facto library for dealing with json in Elixir.
So, I decided to use it and wanted to get familiar with the API. I read some of the code, but ultimately, the task I wanted to achieve was not a complex one:
There current docs seem to cover more complex use cases, dealing with replacing a previous library with this one. But it seems that the most basic cases were left behind unless I missed the docs for them (which is possible).
My steps to find the basic usage were the following:
It was great that it worked, but I bet we can all agree that we want self sufficient docs and not have to go somewhere else other than this repo for usage details
With that in mind, the below would have been sufficient for my use case:
iex(1)> Jason.encode!(%{"age" => 44, "name" => "Steve Irwin", "nationality" => "Australian"})
"{\"age\":44,\"name\":\"Steve Irwin\",\"nationality\":\"Australian\"}"
iex(2)> Jason.decode!("{\"age\":44,\"name\":\"Steve Irwin\",\"nationality\":\"Australian\"}")
%{"age" => 44, "name" => "Steve Irwin", "nationality" => "Australian"}
I'll create a PR updating the docs and you can merge it if you think it is appropriate.
As always, thank you for the work.
https://github.com/michalmuskala/jason/blob/master/lib/encode.ex#L241-L244
It'd be nice for this function to have one more clause:
def key(%{__struct__: module} = struct, escape) do
end
and try to attempt to encode it.
I've found myself in a situation where it would be nice to see if a value is JSON. I do get the error saying invalid JSON ...
so I assume the functionality must be there. Perhaps there already is something in place in that fact.
For now, I've just tried to decode the value if it's successful then assume it is indeed JSON and valid.
It looks like if i try this:
Jason.encode!(%{"severity" => "INFO, "message" => "whoa"})
I get:
"{\"message\":\"whoa\",\"severity\":\"INFO\"}"
But if I do:
Jason.encode!(%{"severity" => "INFO, "xmessage" => "whoa"})
I get
"{\"severity\":\"INFO\",\"xmessage\":\"whoa\"}"
This would suggest the ordering of the keys in the outputted JSON is alphabetical? The only reason I ask is because I want to format my Logs as JSON, and it makes them nice to read if the severity comes first.
Is there anyway to achieve this?
Thanks for this awesome lib!
Support to skip few fields globally if having certain value e.g Ecto.Association.NotLoaded
because if we write customer encoder and return nil for Ecto.Association.NotLoaded
, this is totally incorrect
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.