Bug in offsets readout, unexpected fall back to 0 about unroot.jl HOT 13 CLOSED

Oh well, look at my hack 😆

from unroot.jl.

I just checked with km3io/uproot and there, the offset is continous:

>>> import km3io

>>> f = km3io.OnlineReader("mcv6.1.mupage_10G.si
... rene.jterbr00007209.2548.root")

>>> f.events.snapshot_hits._offsets
array([     0,    151,    198, ..., 783089, 783164, 783229])

from unroot.jl.

Digging deeper, it's related to how we handle multiple entries in fBasketSeek.

For the small file, fBasketSeek only contains a single entry:

julia> f = ROOTFile("/home/tgal/Dev/UnROOT.jl/test/samples/km3net_online.root");

julia> f["KM3NET_EVENT/KM3NET_EVENT/snapshotHits"].fBasketSeek
┌ Warning: Can't automatically create LazyBranch for branch KM3NET_EVENT/KM3NET_EVENT/snapshotHits. Returning a branch object
└ @ UnROOT ~/.julia/packages/UnROOT/o16Nw/src/root.jl:96
10-element Vector{Int64}:
 1606921
       0
       0
       0
       0
       0
       0
       0
       0
       0

Whereas for the larger file we have two entries, in two different parts of the ROOT file:

julia> f["KM3NET_EVENT/KM3NET_EVENT/snapshotHits"].fBasketSeek┌ Warning: Can't automatically create LazyBranch for branch KM3NET_EVENT/KM3NET_EVENT/snapshotHits. Returning a branch object
└ @ UnROOT ~/.julia/packages/UnROOT/o16Nw/src/root.jl:96
10-element Vector{Int64}:
 324654693
 531785725
         0
         0
         0
         0
         0
         0
         0
         0

It's just the stitching of the two different parts, which is buggy, so that will be an easy fix, I think. Let's see... ;)

from unroot.jl.

https://github.com/tamasgal/UnROOT.jl/blob/74735c939e4127ee527270b21be9cd19626566f9/src/root.jl#L280

maybe this? But this should only push the last offset for each basket

from unroot.jl.

I am going through the code. The problem is also the extra entry in the offsets, since it has an additional entry when the basket is changed.

Are you handy with basket option tweaking in ROOT? We need a small test file which uses multiple branches to test this 😕 I am currently looking how to set the basket size but have not found anything useful yet.

from unroot.jl.

if there are 10 elements in a basket, we do expect 11 offsets. Because you index the raw bytes for ith element with offset[i]:offset[i+1] right?

from unroot.jl.

Yes that's true, but I have one more ;)

from unroot.jl.

Just to clarify: the number of offset-elements is correct (11) for the small file with 10 entries. For the large file it's 10412 instead of 10411 (corresponding to 10410 elements).

from unroot.jl.

ah ok, I think the fix is when we dump out the entire branch's raw bytes, for each basket we need to add a "global offset".

and that's probably also where you want to remove the extra offset that is duplicate between two baskets

from unroot.jl.

Yep, we just need to go ahead and count up. I am trying to figure out where to do it, the code has changed a lot 😆

from unroot.jl.

I think just this one function:
https://github.com/tamasgal/UnROOT.jl/blob/74735c939e4127ee527270b21be9cd19626566f9/src/root.jl#L223

if what I imagine happening is causing the issue.

To enable accessing baskets out of order, each baskets return data and local offsets (i.e always starts 0 and have one extra ending), thus indexing and reconstructing within a basket is locally trivial. So if we want a branch's raw bytes, we should trim and stitch them together.

from unroot.jl.

Yes, it should be ripped apart (divide and conquer). For now I'll do a quick fix and later we can discuss how to proceed.

from unroot.jl.

wait, I think we can just fix it by something along the line of:

Seeks = branch.fBasketSeek;
    for i in eachindex(Seeks)
        if  i<length(Seeks) && Seeks[i+1]!=0
            pop!(offsets)
        end
        data, offset = readbasketseek(f, branch, Seeks[i])
        append!(datas, data)
        append!(offsets, offset)
    end

from unroot.jl.