:lady_beetle: ZPAQ Compression/Decompression Broken about lrzip-next HOT 4 CLOSED

It appears that the 7.15 version of zpaq fails when a block of >64MB is sent to the backend on compression. Forcing a limit of 64MB appears to fix the problem. The issue was with compression, but remains unclear.

Taking decompressed data from thread 6
Taking decompressed data from thread 7
Taking decompressed data from thread 8
100%    1363.42 /   1363.42 MB
Closing stream at 156,800,471, want to seek to 156,800,471

Average DeCompression Speed: 20.044MB/s

MD5: 134a2942867678fa1c3d4284c8b738b2
[OK] - 1,429,647,360 bytes                                
Total time: 00:01:08.62

Some additional testing is required.

from lrzip-next.

See new branch zpaq_fix to try out the update. After a review period these changes will be merged. Issue will be reported upstream.

from lrzip-next.

Additional testing has revealed that this statement:

It appears that the 7.15 version of zpaq fails when a block of >64MB is sent to the backend on compression. Forcing a limit of 64MB appears to fix the problem. The issue was with compression, but remains unclear.

Is not entirely correct. The issue seems to be that if the block sent to the ZPAQ backend is greater than than the block size specified, ZPAQ may fail. This does not happen for smaller files that are less than ZPAQ block size. Thus. if the ZPAQ block size is specified as 9, which is 512MB, any block sent must be less than that. There is some code in the code in libzpaq.cpp which defines block size as 1MB << bs - 4096.:

void compress(Reader* in, Writer* out, const char* method,
              const char* filename, const char* comment, bool dosha1) {

  // Get block size
  int bs=4;
  if (method && method[0] && method[1]>='0' && method[1]<='9') {
    bs=method[1]-'0';
    if (method[2]>='0' && method[2]<='9') bs=bs*10+method[2]-'0';
    if (bs>11) bs=11;
  }
  bs=(0x100000<<bs)-4096;

Limiting the block size sent to the ZPAQ backend to 1MB << bs - 4096 seems to keep compression from getting corrupted. Testing is ongoing. Higher block sizes seems to cause intense swap space use.

from lrzip-next.

Looks good. Commits pushed to master and lzma-21.03beta.

from lrzip-next.