Document new WARC fields in 1.x crawler-produced WACZ files about browsertrix HOT 3 OPEN

Are above 3 new warc fields mandatory for modern browserbased replay and are they defacto used in other tools today?

from browsertrix.

Hi @tuehlarsen, longer explanation coming but in short:

• WARC-Resource-Type: Proposal created at iipc/warc-specifications#96; this is used to differentiate resources fetched via JavaScript from those loaded directly in the page, and has other possibilities for future analysis of crawls
• WARC-Page-ID: We added this in Browsertrix to be able to easily associate pages between original crawls and QA replay crawls
• WARC-JSON-Metadata: Proposal created at iipc/warc-specifications#27

None of these additions should cause WARC validation to fail or cause any replay issues. Most software other than Browsertrix will simply ignore the fields, as is suggested in section 5.1 of the WARC 1.1 specification:

Because new fields may be defined in extensions to the core WARC format, WARC processing software shall ignore fields with unrecognized names.

Re: your comment about missing documentation for screenshot and text WARC files, that is noted and should be coming shortly! As of the latest 1.0.0 crawler beta release, these WARCs will also be prefixed if a WARC prefix is specified.

from browsertrix.

I hope it will be more explicitly - as it is of great importance for large older web archives what the new warc fields are for and what they will be used for in the future - it requires some syntax definition and description that can be input to a later iso standardization process..

from browsertrix.