Improve output data compression about gtfs-bench HOT 8 CLOSED

100% agree, tar.xz is perfect!

from gtfs-bench.

The output compression format (zip, tar.bz2 or tar.xz) can be an option of the user interface. If the output is just one format it should be zip as its the most used and compatible.

What do you think @dachafra?

On the other hand, pugz is only an option for gzip, but gzip and zip are different formats. So, pugz is discarded and I have to search for an alternative...

from gtfs-bench.

I wouldn't delegate the compression option to the user sincerely.
Let's prioritize the performance on obtaining the resources instead of the compression format

from gtfs-bench.

btw, I found this: https://github.com/dcwatson/deflate

from gtfs-bench.

btw, I found this: https://github.com/dcwatson/deflate

But that is only the gzip and zip compression algorithm, the interesting thing is if there is a parallel implementation, and for the moment it seams is not possible.

We can move to tar.xz, is compatible with 7-zip on Windows. Is the best option in speed, size and compatibility.

from gtfs-bench.

Some numbers in a machine with 24 computing threads.

Using zip [compression level 9]:

real	5m58.036s
user	5m55.845s
sys	0m2.000s

Using tar + pxz [compression level 1]:

real	0m30.468s
user	8m39.010s
sys	0m18.165s

For a similar output size:

-rw-r--r-- 1 root root 611M Jun 14 11:25 result.tar.xz
-rw-r--r-- 1 root root 652M Jun 14 11:15 result.zip

Better compression ratios can be achieved, but I think the problem is not the size but the compression time.

from gtfs-bench.

No problem with the size! Awesome results, perfect :-D

from gtfs-bench.

tested and working! closing issue :-D

from gtfs-bench.