Comments (8)
Does a3382bb fix the issue?
from lrzip-next.
Does a3382bb fix the issue?
No. New error, and decompression continues to fail..
Fill_buffer stream 1 c_len 5,147,508 u_len 33,550,336 last_head 57,589,094
Starting thread 9 to decompress 5,147,508 bytes from stream 1
internal error: out of thread states.Fatal error - exiting
from lrzip-next.
Can't reproduce:
0 [19:55] Desktop/workspace/[email protected] % src/lrzip-next -B -L1 -vv lrzn.tar
The following options are in effect for this COMPRESSION.
Threading is ENABLED. Number of CPUs detected: 16
Detected 15,534,166,016 bytes ram
Nice Value: 19
Show Progress
Max Verbose
Temporary Directory set as: /tmp/
Compression mode is: BZIP3. LZ4 Compressibility testing enabled
Compression level 1
RZIP Compression level 1
BZIP3 Compression Block Size: 5
MD5 Hashing Used
Heuristically Computed Compression Window: 98 = 9,800MB
Storage time in seconds 1,386,671,352
File size: 31,764,480
Succeeded in testing 31,764,480 sized mmap for rzip pre-processing
Will take 1 pass
Chunk size: 31,764,480
Byte width: 4
Threads reduced to 12
Per Thread Memory Overhead is 201,326,592
Succeeded in testing 2,447,683,584 sized malloc for back end compression
Using up to 12 threads to compress up to 10,485,760 bytes each.
Beginning rzip pre-processing phase
hashsize = 131,072. bits = 17. 2MB
Total: 8% Chunk: 8%
Starting sweep for mask 15
Total: 9% Chunk: 9%
Starting sweep for mask 31
Total: 28% Chunk: 28%
Starting sweep for mask 63
Total: 73% Chunk: 73%
Starting thread 0 to compress 10,485,760 bytes from stream 1
Total: 77% Chunk: 77%
lz4 testing OK for chunk 10,485,760. Compressed size = 63.20% of test size 10,485,760, 1 Passes
Total: 78% Chunk: 78%
Starting bzip3 (bs=5) backend...
Total: 79% Chunk: 79%
Starting sweep for mask 127
Total: 99% Chunk: 99%
87,380 total hashes -- 881 in primary bucket (1.008%)
Malloced 5,178,052,608 for checksum ckbuf
Starting thread 1 to compress 119,370 bytes from stream 0
Starting thread 2 to compress 6,437,089 bytes from stream 1
lz4 testing OK for chunk 119,370. Compressed size = 81.35% of test size 119,370, 1 Passes
Starting bzip3 (bs=5) backend...
lz4 testing OK for chunk 6,437,089. Compressed size = 45.58% of test size 6,437,089, 1 Passes
Starting bzip3 (bs=5) backend...
Writing initial chunk bytes value 4 at 20
Writing EOF flag as 1
Writing initial header at 26
Compthread 0 seeking to 22 to store length 4
Compthread 0 seeking to 26 to write header
Thread 0 writing 5,751,546 compressed bytes from stream 1
Compthread 0 writing data at 39
Compthread 1 seeking to 9 to store length 4
Compthread 1 seeking to 5,751,585 to write header
Thread 1 writing 58,562 compressed bytes from stream 0
Compthread 1 writing data at 5,751,598
Compthread 2 seeking to 35 to store length 4
Compthread 2 seeking to 5,810,160 to write header
Thread 2 writing 1,885,727 compressed bytes from stream 1
Compthread 2 writing data at 5,810,173
MD5: f6b1057e006a7f4863409a0231e2783c
matches=12,284 match_bytes=14,841,631
literals=11,126 literal_bytes=16,922,849
true_tag_positives=25,528 false_tag_positives=68,539
inserts=306,596 match 0.877
lrzn.tar - Compression Ratio: 4.127. bpb: 1.938. Average Compression Speed: 30.000MB/s.
Total time: 00:00:01.53
0 [19:56] Desktop/workspace/[email protected] % src/lrzip-next -d lrzn.tar.lrz -o mtp
Output filename is: mtp
Validating file for consistency...[OK]
100% 30.29 / 30.29 MB0 / 30.29 MB
Average DeCompression Speed: 15.000MB/s
MD5:f6b1057e006a7f4863409a0231e2783c
Output filename is: mtp: [OK] - 31,764,480 bytes
Total time: 00:00:01.41
from lrzip-next.
The internal error that you have posted occurs only if more threads are launched than control->threads
. This should not happen, I assume.
from lrzip-next.
The internal error that you have posted occurs only if more threads are launched than
control->threads
. This should not happen, I assume.
The issue was you did not account for the thread used by rzip. Small changes to
setup_states
lock_state
unlock_state
in for loops.
index 3cd06ec..6460e7a 100644
--- a/src/stream.c
+++ b/src/stream.c
@@ -165,12 +165,12 @@ static int * statequeue = NULL;
static void setup_states(rzip_control * control) {
int i;
- states = malloc(sizeof(struct bz3_state *) * control->threads);
- statequeue = malloc(sizeof(int) * control->threads);
- memset(statequeue, 0, sizeof(int) * control->threads);
+ states = malloc(sizeof(struct bz3_state *) * control->threads + 1);
+ statequeue = malloc(sizeof(int) * control->threads + 1);
+ memset(statequeue, 0, sizeof(int) * control->threads + 1);
if(!states)
fatal("Failed to allocate memory for bzip3 states\n");
- for (i = 0; i < control->threads; i++) {
+ for (i = 0; i < control->threads + 1; i++) {
states[i] = bz3_new((1 << control->bzip3_bs) * ONE_MB);
if(!states[i])
fatal("Failed to allocate %dMB bzip3 state #%d.\n", (1 << control->bzip3_bs), i);
@@ -180,7 +180,7 @@ static void setup_states(rzip_control * control) {
static struct bz3_state * lock_state(rzip_control * control) {
lock_mutex(control, &bz3_statemutex);
int i;
- for (i = 0; i < control->threads; i++) {
+ for (i = 0; i < control->threads + 1; i++) {
if (!statequeue[i]) {
statequeue[i] = 1;
unlock_mutex(control, &bz3_statemutex);
@@ -195,7 +195,7 @@ static struct bz3_state * lock_state(rzip_control * control) {
static void unlock_state(rzip_control * control, struct bz3_state * state) {
lock_mutex(control, &bz3_statemutex);
int i;
- for (i = 0; i < control->threads; i++) {
+ for (i = 0; i < control->threads + 1; i++) {
if (states[i] == state) {
statequeue[i] = 0;
unlock_mutex(control, &bz3_statemutex);
Now we can move forward. However, I do not understand why you have to lock states for bzip3 when each thread is already locked. Can't the functions bzip3_compress_buf
and bzip3_decompress_buf
have their own local state structures - one for each thread? This would greatly simplify the codebase and keep each compress
and decompress
function similar.
from lrzip-next.
@pete4abw This fix is not correct. Consider parenthesising (control->threads + 1)
, otherwise multiplication will be performed first, leading to UB on access to the last element.
from lrzip-next.
@pete4abw This fix is not correct. Consider parenthesising
(control->threads + 1)
, otherwise multiplication will be performed first, leading to UB on access to the last element.
Yup. Fixed. However, I did not notice any problem without the () which I find odd. Anyway, as I wrote earlier, it would be better if we could preserve the handling of all the _compress_buf
and _decompress_buf
without the bzip3 helper functions.
Will keep testing.
from lrzip-next.
According to the C standard, the chunk of memory allocated by malloc()
is only guaranteed to be as many bytes as you ask for, but in practice most implementations give you a little extra for padding or the purposes of the memory manager.
from lrzip-next.
Related Issues (20)
- ZPAQ Compression issue when lrizip-next block size > zpaq block size HOT 3
- :lady_beetle: linker warning: missing .note.GNU-stack section implies executable stack HOT 2
- Found typo in viewing compression info on a lrz file HOT 1
- :lady_beetle: lrzip-next segfaults when STDIN + -o filename used. HOT 2
- :lady_beetle: [Certain files compressed with ZPAQ fail to compress successfully] HOT 5
- :lady_beetle: [Slower than lrzip, same arguments, stuck at 99%] HOT 4
- :lady_beetle: [Broken Pipe at 55% on L9] HOT 21
- Huanglei3 reported a bug in lrzip that is not duplicated here HOT 3
- Read zero page memory HOT 1
- :bulb: Have lzma backend respect -p# option HOT 9
- :bulb: possible precomp support HOT 2
- :lady_beetle: Decompress/Test/Info fails with --bzip2, gzip, lzo
- :lady_beetle: Losing on ISO/Zip on rare cases HOT 11
- From lrzip: lrzip 0.631 occasionally hangs forever HOT 5
- Enhance lzo compression by using all 5 methods HOT 2
- :lady_beetle: double free detected in tcache 2 HOT 1
- :lady_beetle: zstd compression fails when all blocks are incompressible HOT 1
- :lady_beetle: fake_mremap does not compile HOT 3
- From lrzip: Suggestion to support encrypted Stdin/Stdout when password provided on command line #251 HOT 1
- :bulb: Allow PROCESSORS to be set in lrzip.conf file HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from lrzip-next.