:bulb: Improve stream_bufsize for stream.c:open_stream_out about lrzip-next HOT 5 CLOSED

Discussion when LZMA or ZPAQ used:

Ramsize = total system ram
Usable ram = Ram size / 3 or 6 depending on whether or not STDIN/OUT is used
Stream out Limit = Usable ram / 2 (when compression is used)
Overhead = ram used by LZMA or ZPAQ (only applicable when LZMA or ZPAQ used)
Threads * Overhead has to be less than Limit

Instead of reducing Limit, we reduce threads and then overhead size.

Stream bufffer size should be Threads * Overhead, not Limit / Threads.

The goal is to have the largest possible buffer passed to backend.

from lrzip-next.

When lzma levels are low, the dictionary size is really low, so that passing such a small buffer is impractical. For example, at Level 1, the lzma dictSize is only 256KB. At Level 3, just 4MB. So, control->overhead just won't do it. Now, a focus will change to

MAX(limit/threads, control->overhead)

Work continues...

The same may be true with ZPAQ.

from lrzip-next.

The new stream_buffsize_fix branch is live. A test suite was run on a 22GB Android source file with lrzip and lrzip-next for all compression levels. The two takeaways were

1. lrzip-next compressed better
2. lrzip-next was slower

The following table explains.

Source file size: 22,186,711,040
Host machine 16GB ram, NVMe drives

from lrzip-next.

Next step to optimize is this. If we know the size of the file being compressed, and it is less than the chunk limit we could possibly make a better computation for overhead size comparison and threads used.

from lrzip-next.

All done. Merged into lzma-sdk-21.04beta.

from lrzip-next.