Comments (10)
I now noticed that, If I do memset with 0 for dest buffer, this is not
happening.
Memset is a must? or anyway to skip that to get better speed?
Original comment by [email protected]
on 23 Jun 2012 at 3:53
from lz4.
Data corruption is always considered critical. I'll look into it.
Original comment by [email protected]
on 23 Jun 2012 at 10:06
- Changed state: Accepted
from lz4.
In your examples, the corrupted byte is always the 3rd one.
Is that happening at this position in the file ? or is it just the 3rd position
on the sample line ?
Another question : is the corruption happening only when compressing with
LZ4HC, or does it also happen on the same file when compressing with LZ4 ?
Last point : is it possible to get access to the file, or the portion of file
where the problem happens ? This would help to reproduce the issue.
Rgds
Original comment by [email protected]
on 23 Jun 2012 at 10:11
from lz4.
When I tried yesterday, the position was always third one from the offset of
that line. (offset is the 4 hex number before colon ":" on that line)
But looks like that is not always third one, when I tried today also appearing
on 7th postion of that line.
--- lz4hc compressed ---
15C5 99D0: 4D F6 C1 10 74 07 38 3C 00 00 C0 EB 46 F6 C1 84 M...t.8< ....F...
15C5 99D0: 4D F6 C1 10 74 07 B8 3C 00 00 C0 EB 46 F6 C1 84 M...t..< ....F...
1747 FAD0: 74 0D 8B 07 6A 01 8B CF FF 10 EB 03 88 5D FF 8D t...j... .....]..
1747 FAD0: 74 0D 8B 07 6A 01 0B CF FF 10 EB 03 88 5D FF 8D t...j... .....]..
3CBF 3D20: 07 50 03 C1 04 51 81 C2 A8 00 00 00 52 E8 6E C3 .P...Q.. ....R.n.
3CBF 3D20: 07 50 83 C1 04 51 81 C2 A8 00 00 00 52 E8 6E C3 .P...Q.. ....R.n.
412F 3C20: 00 00 8B C3 5B 59 5D C3 55 8B EC 81 C4 F0 FB FF ....[Y]. U.......
412F 3C20: 00 00 0B C3 5B 59 5D C3 55 8B EC 81 C4 F0 FB FF ....[Y]. U.......
4C48 5E60: 5F 5F 71 29 0A 20 20 20 20 7B 20 72 65 74 75 72 __q). { retur
4C48 5E60: 5F 5F 79 29 0A 20 20 20 20 7B 20 72 65 74 75 72 __y). { retur
4ED6 EEE0: 8D EB 31 FF 85 C0 0F 85 1C 02 00 00 FF 75 3C E8 ..1..... .....u<.
4ED6 EEE0: 8D EB B1 FF 85 C0 0F 85 1C 02 00 00 FF 75 3C E8 ........ .....u<.
--- lz4 compressed (1st try) ---
0841 58D0: EF 45 35 1C 6F 28 40 82 EF 00 ED E7 B1 7E 4B 83 .E5.o(@. .....~K.
0841 58D0: EF 45 35 1C 6F 28 48 82 EF 00 ED E7 B1 7E 4B 83 .E5.o(H. .....~K.
529F 9B10: E3 36 EF 36 0B 37 14 37 6B 37 ED 37 2D 38 8B 38 .6.6.7.7 k7.7-8.8
529F 9B10: E3 36 EF 36 0B 37 10 37 6B 37 ED 37 2D 38 8B 38 .6.6.7.7 k7.7-8.8
--- lz4 compressed (2nd try) ---
1447 99D0: 10 18 D7 82 C4 36 9B 5B 49 BB C1 9F 6F 23 68 28 .....6.[ I...o#h(
1447 99D0: 10 18 D7 82 C4 36 13 5B 49 BB C1 9F 6F 23 68 28 .....6.[ I...o#h(
28BF BB90: 08 81 09 84 00 0A 07 84 84 84 87 87 87 86 86 86 ........ ........
28BF BB90: 08 81 09 84 00 0A 87 84 84 84 87 87 87 86 86 86 ........ ........
41ED FE20: F9 F9 F2 FD FE FE FA FA FD FA FA FA FA FA FA FA ........ ........
41ED FE20: F9 F9 FA FD FE FE FA FA FD FA FA FA FA FA FA FA ........ ........
I have deleted the original file because so I extracted with memset 0 and
recompressed with lz4, tested above two. But still happening when extracting
without memset.
The file is the virtualbox vdi file, includes running windows XP, some
softwares inside. I think I am not supposed to post downloadable link online.
But let me know if you really need it. I could upload and send the link to your
private email address. Is the email you registered here is private one?
Regards,
Original comment by [email protected]
on 24 Jun 2012 at 9:36
from lz4.
OK, thanks for the traces.
If i do understand correctly :
- Issue happens on the decoder side, whatever the compression algorithm (lz4 -
lz4hc)
- Issue is random : you seem to have different errors at different positions
between each try.
- Corrupted bytes are isolated , there is no long trails of errors. This part
is really uncommon. I see no valid reason for this to happen with the LZ4
algorithm : it should quickly propagate the error further into the file.
- Even more precisely, the error seems to concern one bit. Listing your
examples :
38 -> B8 (bit 7)
8B -> 0B (bit 7)
03 -> 83 (bit 7)
and so on.
It means the corrupted byte does not seem completely random.
This is unexpected : LZ4 is byte-oriented, not bit-oriented. And there is no
masking anywhere during decoding. There is no reason it would wrongly output
just one bit : the wrong byte should be completely random.
OK, so since memset() seems to correct the issue, i'm starting to wonder if it
could br related to allocation mechanism : since with memset(), memory is
really allocated beforehand, while without it, memory is merely "reserved", to
be allocated "just in time" later on. Maybe at this stage, something may not
happen properly. Maybe a race condition ?
Last point : is this issue reproducible on another machine ? Could it be
hardware-related ?
Regarding the sample file : yes, having a sample which allows to reproduce the
issue would help a lot to understand it. You don't need the full file, just a
small sample of it which triggers the issue.
But, if the problem is random, this might be difficult...
Original comment by [email protected]
on 25 Jun 2012 at 3:57
from lz4.
>> This is unexpected : LZ4 is byte-oriented, not bit-oriented.
Thanks, I think that answered a lot. Need to doubt my machine now.
I was originally thought some masking happened with dest buffer. But since lz4
is not using masking during decoding this shouldn't happened, and actually I
tried to dump memory before decompress without memset yesterday, and I noticed
the whole memory was already all-zeros.
>> it could br related to allocation mechanism : since with memset(), memory is
really allocated beforehand, while without it, memory is merely "reserved", to
be allocated "just in time" later on
I see, that make sense.
>> Maybe a race condition ?
>> Could it be hardware-related ?
Could be, and may be memory related issue, and I am extracting the file into
dynamically expanded ramdisk actually. I thought that shouldn't be problem, so
I was not mentioning it.
>> is this issue reproducible on another machine ?
I don't have another machine with enough memory to test same way at the moment.
I will try to test some more, and I will report the progress. Probably you may
close the issue once confirmed.
Rgds,
Original comment by [email protected]
on 26 Jun 2012 at 1:05
from lz4.
You're welcomed.
Please keep us posted about your progresses.
Rgds
Original comment by [email protected]
on 26 Jun 2012 at 7:38
from lz4.
I tested without using ramdisk and but unfortunately results were same, and its
just random. Also tested on another kernel on same machine, but same results.
So, may be this is just my machine problem.
Rgds,
Original comment by [email protected]
on 27 Jun 2012 at 12:14
from lz4.
Maybe, possibly.
At least, it's not ramdisk nor kernel related.
I would feel better if the problem could prove not being repeatable on another
machine.
Rgds
Original comment by [email protected]
on 27 Jun 2012 at 2:24
from lz4.
Closed.
The issue seems related to Hardware problems.
Don't hesitate to open it again if the above hypothesis prove wrong.
Original comment by [email protected]
on 10 Jul 2012 at 10:55
- Changed state: Invalid
from lz4.
Related Issues (20)
- Clang compiler warnings HOT 6
- LZ4F_compressBound() using null PreferencesPtr HOT 3
- lz4io license HOT 6
- Commandline lz4 stops when decompressing a concatenated lz4 file representing an empty file HOT 4
- Kernel wont boot with CONFIG_DYNAMIC_DEBUG=y HOT 2
- Bad alignment detection & setting on Sparc Solaris HOT 2
- lz4 r124 breaks ABI without bumping soname HOT 22
- Do not call sudo in Makefile HOT 3
- LZ4 HC r126 crashes when compiled with GCC 4.9 on highest optimization settings HOT 1
- lz4frame should correctly handle skipped frames HOT 4
- LZ4 cli invocation should match other compressors for non-option arguments HOT 14
- sudo make install does not cp files to /usr/local HOT 6
- lz4 auto-decompress does not work HOT 3
- Force should pass thru bytes unchanged HOT 5
- lz4 cli should support sparse file HOT 22
- decompression same file on different platforms HOT 3
- Static code analysis warnings HOT 10
- Google Code will close HOT 3
- [Query]Can LZo algorithm optimized for ARM NEON (SIMD instructions). HOT 3
- Potential Size Extension Problem in LZ4F_readLE64 HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from lz4.