Feature request: add xxhash for use with LZ4 about cramjam HOT 4 CLOSED

Initially, I'm thinking this is interesting for a separate project (hashing as to cramjam's compression orientation), however just from glancing at xxhash briefly, I'm not sure it fits into the scope/genre of cramjam.

I'm also a bit confused how python's lz4 relates to xxhash? (at least for how you get it) Scanning the docs I don't see any API for it. Is it python's lz4 ships the xxhash C lib w/ it?

Anyhow, I think getting a xxhash impl going and running on pyodide should be pretty okay just don't have the free time available for it. On the off chance you've got some funding available, I'd be happy to do this under a contract agreement. (feel free to email me if this is the case and we can go from there) Otherwise I may do something much later when time is more present. :)

from cramjam.

The reason that I'm interested is because we have a file format that always precedes an LZ4-compressed block of data with an xxhash checksum, so LZ4 and xxhash are linked as dependencies: if we need LZ4, then we need xxhash.

https://github.com/scikit-hep/uproot5/blob/0f323c093448cc739d80b5a50dc08c9873cd35ff/src/uproot/compression.py#L425-L441

I'm not entirely sure why xxhash is linked to LZ4 specifically—maybe the other compressors have built-in checksums and LZ4 doesn't, so xxhash arbitrarily chosen to fill that gap? When I saw that the C implementation (not Python) of liblz4 has a header file called xxhash, I thought that explained the connection: these two algorithms have a common origin, but they've been packaged separately in Python.

Another avenue I can try is to see if the Python xxhash library can get added to Pyodide. Maybe it's easier for xxhash to add a meta.yaml build script to Pyodide than for you to port the C code of the xxhash algorithm to Rust? From the perspective of a dependent library that wants to run on JupyterLite, either would work for me, and now that I'm talking myself into it, it sounds like adding a build target is easier than porting an algorithm between languages with sufficient testing.

from cramjam.

To cross-link these, here's the request I made at python-xxhash: ifduyue/python-xxhash#65

from cramjam.

I agree, adding to python-xxhash would be the most beneficial. Anyway thanks for the additional context and information. Closing this as I think it's out of scope for cramjam.

from cramjam.