Trouble unzipping `sitemap.xml` (`zlib: incorrect header check`) about crawlee HOT 5 CLOSED

Looks like the sitemap you reference points to another - https://www.paypal-community.com/sitemap_threads.xml.gz - which is not gzipped, despite its name. Why would anyone do this is beyond me. I guess we could add an option to Sitemap.load to override the type 🤷... Trying to recover automatically seems futile to me.

EDIT: the server uses in-transit gzip encoding, which is however a different thing than what the .xml.gz extension is supposed to mean

from crawlee.

Can't we detect this based on some initial bytes? Hard to trust extensions for anything these days :D

edit: something like this

In the modified code, we are checking for the GZip file format by comparing the first 2 bytes of the file with the expected byte sequence ({&H1F, &H8B}). GZip files typically start with these two bytes.

from crawlee.

Can't we detect this based on some initial bytes? Hard to trust extensions for anything these days :D

It's gonna be tricky since we use streams, but feasible, I guess.

from crawlee.

You can use file-type (example with streams here), in WCC it worked pretty well. You can even pipe through it 👀

from crawlee.

More WCC users are complaining about this. Do we know how to approach this issue yet?

from crawlee.