Unsupported compression type 'L4' about unroot.jl HOT 14 CLOSED

I guess I will simply dump the raw uncompressed data as a binary file and then fire up the hex editor to inspect it. Additionally I'll also try other LZ4 decompressor libraries (e.g. in Python) to see if it's maybe a problem in CodecLz4.jl. At least uproot is able to read the file...

from unroot.jl.

nvm, I got it working, pushing now!

from unroot.jl.

Yep thanks for the issue! LZ4 is on the todo list, but I have never worked with ROOT files using that compression, so it was not a high priority.

With your file, I can now have a look but in principle it should be fairly easy, since luckily most of the compression codecs in Julia are based on https://github.com/JuliaIO/TranscodingStreams.jl which means that the API is the same!

from unroot.jl.

I forgot, could you upload a small sample of that file @bernd1995 ?

from unroot.jl.

Of course.
I added two files with the same content.
test_compression.zip

test_comp.root is the compressed and unreadable version, test_uncomp.root is the readable file.

from unroot.jl.

Perfect, thanks!

from unroot.jl.

I tried out my first idea yesterday, to just add the package https://github.com/JuliaIO/CodecLz4.jl and in the function Streamers(io) in streamers.jl change lines 84-88 (streamers.jl#L88) to

if String(compression_header.algo) == "ZL"
    stream = IOBuffer(read(ZlibDecompressorStream(io), tkey.fObjlen))
elseif String(compression_header.algo) == "L4"
    stream = IOBuffer(read(LZ4SafeDecompressorStream(io),tkey.fObjlen))
else
    error("Unsupported compression type '$(String(compression_header.algo))'")
end

However, it seems that there is some different behavior in the read() function for the ::TranscodingStreams.TranscodingStream{LZ4SafeDecompressor,IOStream} datatype than for the ::TranscodingStreams.TranscodingStream{ZlibDecompressor,IOStream} because I get the following error message

f=ROOTFile("/hosts/nashome/riederer_bernd/test_comp.root")
ERROR: LZ4_decompress_safe_continue: Decompression failed.
Stacktrace:
 [1] changemode!(::TranscodingStreams.TranscodingStream{LZ4SafeDecompressor,IOStream}, ::Symbol) at /hosts/nashome/riederer_bernd/.julia/packages/TranscodingStreams/MsN8d/src/stream.jl:717
 [2] callprocess(::TranscodingStreams.TranscodingStream{LZ4SafeDecompressor,IOStream}, ::TranscodingStreams.Buffer, ::TranscodingStreams.Buffer) at /hosts/nashome/riederer_bernd/.julia/packages/TranscodingStreams/MsN8d/src/stream.jl:649
 [3] fillbuffer(::TranscodingStreams.TranscodingStream{LZ4SafeDecompressor,IOStream}; eager::Bool) at /hosts/nashome/riederer_bernd/.julia/packages/TranscodingStreams/MsN8d/src/stream.jl:577
 [4] fillbuffer at /hosts/nashome/riederer_bernd/.julia/packages/TranscodingStreams/MsN8d/src/stream.jl:564 [inlined]
 [5] eof(::TranscodingStreams.TranscodingStream{LZ4SafeDecompressor,IOStream}) at /hosts/nashome/riederer_bernd/.julia/packages/TranscodingStreams/MsN8d/src/stream.jl:188
 [6] readbytes!(::TranscodingStreams.TranscodingStream{LZ4SafeDecompressor,IOStream}, ::Array{UInt8,1}, ::Int32) at /hosts/nashome/riederer_bernd/.julia/packages/TranscodingStreams/MsN8d/src/stream.jl:371
 [7] read(::TranscodingStreams.TranscodingStream{LZ4SafeDecompressor,IOStream}, ::Int32) at ./io.jl:941
 [8] UnROOT.Streamers(::IOStream) at /hosts/nashome/riederer_bernd/.julia/dev/UnROOT/src/streamers.jl:87
 [9] ROOTFile(::String) at /hosts/nashome/riederer_bernd/.julia/dev/UnROOT/src/root.jl:39
 [10] top-level scope at none:1

from unroot.jl.

Thanks for the efforts Bernd! Yes I spent two hours yesterday and also could not decompress it.

The LZ4 algorithm requires also the uncompressed size, as far as I understood so I also had to pass tkey.fObjlen to the compressor stream. But still, I get Decompression failed just like you.

I'll dig deeper today. Btw. the compression part needs to be tweaked anyways since currently it's more or less hardcoded to a single algorithm. For this I played around with a new struct to make it easier, but it's not finished yet.

from unroot.jl.

Btw. here is the line which I meant: https://github.com/tamasgal/UnROOT.jl/blob/edb08617293e2f248439e40e4952793ada2e16cd/src/streamers.jl#L89

However it does not work yet, as mentioned 😉

from unroot.jl.

@Moelf figured it out! I think this should work now. @bernd1995 could you give it a try with the lastes master? v0.1.7 should also be ready within the next hour...

See #19

from unroot.jl.

to be clear, you still can't read data yet, *I think it is because baskets in branch are interlaced with stuff that are not compressed by LZ4 (ROOT's stuff? Tamas can tell me how does that work).

For posterity, LZ4, in block mode, needs to know how long the data is after decompression. AND, there's a 8 bytes checksum (uproot reference) at the beginning of every "block", which is what you guys were missing I think. But also may come to play in branch baskets...

from unroot.jl.

Ah ok, sorry for the confusion, at least we have big first leap ;) I'll check it out!

from unroot.jl.

Hey nice work still 😃
I'll try to have a look at it tomorrow but can't promise anything here.

from unroot.jl.

oh wow 😆

from unroot.jl.