Collection loading stalls. about replayweb.page HOT 6 CLOSED

Is there a solution for this?
I use(d) grab-site to create large, singular warc files (with uncompressed .cdx) and just noticed that ReplayWeb is not able to open these. Previously I only tested with smaller archives.
Is there a tool to convert my existing warc archive to a WACZ collection without scraping again?

edit: currently reading the linked repo. Will try this
edit: There's now a tool for such a conversion in the linked repo. Great!

from replayweb.page.

Yes, would recommend using WACZ for larger warcs due to browsers issue having to load the entire file into memory at once. Firefox I know still seems to lock up randomly when streaming a larger file.
Perhaps grab-site can have an option to generate wacz, I'll suggest that there.

from replayweb.page.

Yeah, at this scale (30GB+), it won't be able to load it all with just the WARC.
It needs to be converted into a WACZ collection with a compressed index, and then it can load the collection on-demand.
I don't quite have the tools ready to do this, but the idea is that it could take a pywb collection and create a compressed .wacz file as per: https://github.com/webrecorder/web-archive-collection-format

I'll try to have a skeleton of a tool that does this (for the current spec) fairly soon, if you want to try it out.

from replayweb.page.

Ah, sorry I should have clarified: I packaged the collection as WACZ before attempting to load it into replayweb.page.

from replayweb.page.

Ah ok! But just using the plain, uncompressed .cdx, right? WACZ supports both compressed and uncompressed
The compression part is rather straightforward, just need to have a script that does it.. I've been using the one on webarchive-indexing but its a bit old..

from replayweb.page.

The latest version includes some fixes to prevent timing out when loading large WARCs (1.3.11), hopefully even on Firefox, so will close this for now. WACZ is still recommended for large WARCs, but I think this is now working as best as possible, given that the entire WARC must be loaded

from replayweb.page.