Evaluate efficiency of zsync2 about appimageupdate HOT 3 OPEN

Chopping the file in chunks more intelligently could possibly help, e.g., after each file inside the squashfs. Also the different compressors might influence this. Let's keep in mind that we might want to use the Zstandard compressor now that it is available.

This area of research is especially important when combined with p2p, because if we get chunks with checksums that are the same between different files, then this helps the p2p performance greatly... so let's see if we can get some people from #ipfs and #ipfs-dev interested in this, too...

from appimageupdate.

@probonopd that's what I thought, too, I've commented in the other issue. This is all about playing with the parameters for both mksquashfs and zsync2 for now. But it should be done properly.

If it turns out that compression ruins block alignment (although I think the only sane way they implemented it is that they did it file wise), we'd have to change it to compressing files rather than the image or chunks with a size higher than the block sizes.

For now, I can tell that the block sizes of mksquashfs and zsync2 differ right now. First approach will be to equalize them, because using a block size lower than the one used for the squashfs image for zsync2 doesn't add anything but costs performance and adds bloat (i.e., additional hashes, by the factor (mksquashfs block size) / (zsync2 block size)), so that's the first thing to optimize. But, please keep in mind that in order to get meaningful results, I wouldn't change either option before performing the evaluations. I might be wrong as well.

from appimageupdate.

Maybe we can get @whyrusleeping's opinion on this one, too.

He suggested that we may need to make the ipfs chunker aware of the compressed file format.

Probably using the right type of compression for the AppImage could help a lot, so let's understand

• facebook/zstd#395 (Splittable/Seekable Format with Jump Table)

Reference:
AppImage/AppImageKit#175 (comment)

from appimageupdate.