use more portable/stable serialization/deserialization about benchmarktools.jl HOT 7 CLOSED

While JLD is valid HDF5, I've been told JLD is painfully structured for use in other languages.

Would be interesting to know more detail. Perhaps post an issue over at JLD?

from benchmarktools.jl.

Would be interesting to know more detail. Perhaps post an issue over at JLD?

I would if it was my own complaint - I've never personally attempted to use JLD outside Julia. This was an issue somebody relayed to me in a conversation about my usage of JLD for benchmark data.

from benchmarktools.jl.

I mean, JSON is probably going to be your most natural format given the nested nature of the BenchmarkTools/groups datastructures, but I'm partial to CSV since it's easier to then ingest/visualize the data.

from benchmarktools.jl.

This was an issue somebody relayed to me in a conversation about my usage of JLD for benchmark data.

Feel free to encourage them to chime in, then.

from benchmarktools.jl.

While I'm sorry about your troubles with JLD, do keep in mind that any serializer other than the one in Base (which doesn't not guarantee backwards compatibility) may encounter trouble tracking julia. For example, the recent breakage came from how SimpleVector gets (de)serialized (due to the change in an import in Base). With JSON, here's what you get currently:

julia> v = Core.svec(1, 2, "foo")
svec(1,2,"foo")

julia> JSON.json(v)
"{\"length\":3}"

julia> JSON.parse(JSON.json(v))
Dict{String,Any} with 1 entry:
  "length" => 3

This may not be exactly what you'd like, either. In a sense, any sufficiently-capable serializer is also guaranteed to be "fragile" with respect to changes in julia, since it will probably rely on julia for some of its mechanisms.

Perhaps the best solution is to couple tests of our various serializers to changes in julia.

from benchmarktools.jl.

Now that I've looked more closely, I agree with the idea of using a different serializer at least for BaseBenchmarks. The primary purpose is to support testing master, and master will frequently break JLD because JLD leverages a lot of low-level type machinery. If you write out the serialization manually using the exported features of Julia, it will be more robust against changes to Base.

from benchmarktools.jl.

Since this issue came up again (kind of) in JuliaIO/JLD.jl#196, it'd be a good time to reiterate how nice it would be to have a simple JSON (de)serializer in BenchmarkTools so we could get rid of the JLD dependency (which has caused pain for non-Nanosoldier users anyway, see #53).

Here's the serialization format I came up with many moons ago, should someone want to implement it: https://github.com/JuliaCI/BenchmarkTools.jl/blob/f9376e6ffafa0c491a7906ab3b3ed4656b24f92d/src/serialization.jl. I thought I had already shared this format here or in JuliaCI/Nanosoldier.jl#36, but it looks like I had only shared it with folks on Slack.

from benchmarktools.jl.