LZ4_uncompress output differ on big data? about lz4 HOT 10 CLOSED

I now noticed that, If I do memset with 0 for dest buffer, this is not 
happening.
Memset is a must? or anyway to skip that to get better speed?

Data corruption is always considered critical. I'll look into it.

In your examples, the corrupted byte is always the 3rd one.

Is that happening at this position in the file ? or is it just the 3rd position 
on the sample line ?

Another question : is the corruption happening only when compressing with 
LZ4HC, or does it also happen on the same file when compressing with LZ4 ?

Last point : is it possible to get access to the file, or the portion of file 
where the problem happens ? This would help to reproduce the issue.

Rgds

When I tried yesterday, the position was always third one from the offset of 
that line. (offset is the 4 hex number before colon ":" on that line)

But looks like that is not always third one, when I tried today also appearing 
on 7th postion of that line.

--- lz4hc compressed ---

15C5 99D0: 4D F6 C1 10 74 07 38 3C  00 00 C0 EB 46 F6 C1 84  M...t.8< ....F...  
15C5 99D0: 4D F6 C1 10 74 07 B8 3C  00 00 C0 EB 46 F6 C1 84  M...t..< ....F...  

1747 FAD0: 74 0D 8B 07 6A 01 8B CF  FF 10 EB 03 88 5D FF 8D  t...j... .....]..  
1747 FAD0: 74 0D 8B 07 6A 01 0B CF  FF 10 EB 03 88 5D FF 8D  t...j... .....]..  

3CBF 3D20: 07 50 03 C1 04 51 81 C2  A8 00 00 00 52 E8 6E C3  .P...Q.. ....R.n.  
3CBF 3D20: 07 50 83 C1 04 51 81 C2  A8 00 00 00 52 E8 6E C3  .P...Q.. ....R.n.  

412F 3C20: 00 00 8B C3 5B 59 5D C3  55 8B EC 81 C4 F0 FB FF  ....[Y]. U.......  
412F 3C20: 00 00 0B C3 5B 59 5D C3  55 8B EC 81 C4 F0 FB FF  ....[Y]. U.......  

4C48 5E60: 5F 5F 71 29 0A 20 20 20  20 7B 20 72 65 74 75 72  __q).     { retur  
4C48 5E60: 5F 5F 79 29 0A 20 20 20  20 7B 20 72 65 74 75 72  __y).     { retur  

4ED6 EEE0: 8D EB 31 FF 85 C0 0F 85  1C 02 00 00 FF 75 3C E8  ..1..... .....u<.  
4ED6 EEE0: 8D EB B1 FF 85 C0 0F 85  1C 02 00 00 FF 75 3C E8  ........ .....u<.  

--- lz4 compressed (1st try) ---

0841 58D0: EF 45 35 1C 6F 28 40 82  EF 00 ED E7 B1 7E 4B 83  .E5.o(@. .....~K.  
0841 58D0: EF 45 35 1C 6F 28 48 82  EF 00 ED E7 B1 7E 4B 83  .E5.o(H. .....~K.  

529F 9B10: E3 36 EF 36 0B 37 14 37  6B 37 ED 37 2D 38 8B 38  .6.6.7.7 k7.7-8.8  
529F 9B10: E3 36 EF 36 0B 37 10 37  6B 37 ED 37 2D 38 8B 38  .6.6.7.7 k7.7-8.8  

--- lz4 compressed (2nd try) ---

1447 99D0: 10 18 D7 82 C4 36 9B 5B  49 BB C1 9F 6F 23 68 28  .....6.[ I...o#h(  
1447 99D0: 10 18 D7 82 C4 36 13 5B  49 BB C1 9F 6F 23 68 28  .....6.[ I...o#h(  

28BF BB90: 08 81 09 84 00 0A 07 84  84 84 87 87 87 86 86 86  ........ ........  
28BF BB90: 08 81 09 84 00 0A 87 84  84 84 87 87 87 86 86 86  ........ ........  

41ED FE20: F9 F9 F2 FD FE FE FA FA  FD FA FA FA FA FA FA FA  ........ ........  
41ED FE20: F9 F9 FA FD FE FE FA FA  FD FA FA FA FA FA FA FA  ........ ........  

I have deleted the original file because so I extracted with memset 0 and 
recompressed with lz4, tested above two. But still happening when extracting 
without memset.

The file is the virtualbox vdi file, includes running windows XP, some 
softwares inside. I think I am not supposed to post downloadable link online. 
But let me know if you really need it. I could upload and send the link to your 
private email address. Is the email you registered here is private one?

Regards,

OK, thanks for the traces.
If i do understand correctly :

- Issue happens on the decoder side, whatever the compression algorithm (lz4 - 
lz4hc)

- Issue is random : you seem to have different errors at different positions 
between each try.

- Corrupted bytes are isolated , there is no long trails of errors. This part 
is really uncommon. I see no valid reason for this to happen with the LZ4 
algorithm : it should quickly propagate the error further into the file.

- Even more precisely, the error seems to concern one bit. Listing your 
examples :
38 -> B8 (bit 7)
8B -> 0B (bit 7)
03 -> 83 (bit 7)
and so on.
It means the corrupted byte does not seem completely random.
This is unexpected : LZ4 is byte-oriented, not bit-oriented. And there is no 
masking anywhere during decoding. There is no reason it would wrongly output 
just one bit : the wrong byte should be completely random.


OK, so since memset() seems to correct the issue, i'm starting to wonder if it 
could br related to allocation mechanism : since with memset(), memory is 
really allocated beforehand, while without it, memory is merely "reserved", to 
be allocated "just in time" later on. Maybe at this stage, something may not 
happen properly. Maybe a race condition ?

Last point : is this issue reproducible on another machine ? Could it be 
hardware-related ?


Regarding the sample file : yes, having a sample which allows to reproduce the 
issue would help a lot to understand it. You don't need the full file, just a 
small sample of it which triggers the issue. 
But, if the problem is random, this might be difficult...

>> This is unexpected : LZ4 is byte-oriented, not bit-oriented.
Thanks, I think that answered a lot. Need to doubt my machine now.

I was originally thought some masking happened with dest buffer. But since lz4 
is not using masking during decoding this shouldn't happened, and actually I 
tried to dump memory before decompress without memset yesterday, and I noticed 
the whole memory was already all-zeros.

>> it could br related to allocation mechanism : since with memset(), memory is 
really allocated beforehand, while without it, memory is merely "reserved", to 
be allocated "just in time" later on
I see, that make sense.

>> Maybe a race condition ?
>> Could it be hardware-related ?
Could be, and may be memory related issue, and I am extracting the file into 
dynamically expanded ramdisk actually. I thought that shouldn't be problem, so 
I was not mentioning it.

>> is this issue reproducible on another machine ?
I don't have another machine with enough memory to test same way at the moment.

I will try to test some more, and I will report the progress. Probably you may 
close the issue once confirmed.

Rgds,

You're welcomed.
Please keep us posted about your progresses.

Rgds

I tested without using ramdisk and but unfortunately results were same, and its 
just random. Also tested on another kernel on same machine, but same results.
So, may be this is just my machine problem.

Rgds,

Maybe, possibly.
At least, it's not ramdisk nor kernel related.
I would feel better if the problem could prove not being repeatable on another 
machine.

Rgds

Closed.
The issue seems related to Hardware problems.
Don't hesitate to open it again if the above hypothesis prove wrong.