Building on the shared kevals library, add hooks to pull data from Kafka or local files and push to a crawl log database.

For crawl streams from Kafka, some field mapping is required to map Heritrix's fields to the warcprox ones:

• size > wire_bytes
• mimetype > content_type
• seed > source

Also:

• annotations and extra_info need mapping to string arrays
• A warc_writer field needs to be added to mark files as from heritrix3 or warcprox.
• The code from anjackson/ukwa-services@4c656a5 needs to be moved over into this codebase or https://github.com/ukwa/kevals as appropriate.

We could use https://wikitech.wikimedia.org/wiki/Event_Platform/EventStreams and make a client to the page-level URL changes stream, and look for UK URLs and domains, and feed them into the crawl input stream.

This line does not report the right queue/topic:

It should use this.queue rather than a hard-coded value.

To avoid issues like ukwa/ukwa-heritrix#76, we could:

• When storing in Kafka, key the launch requests by distinct seed/host configuration.
• Switch to compacted Kafka so only the latest crawl spec. is kept per unique site key.
• Ensure Heritrix always reads the whole Kafka topic, thus reconstructing the up-to-date configuration.#

Cleanest option would be to gather seeds together and assemble an overall launch spec per host, then change how Heritrix processes that to set up the queue config and then enqueue all the seeds with their launchTimestamp used to reflect the most recent launches.

Should we add a FastAPI module so the crawler can have a REST API for launching a crawl. Possibly also for managing scope and seeds files? This could also expose crawl stats etc. The overall approach would be to avoid forcing services that interact with the crawler having to go through Kafka, at least for crawl launches.

This could be made consistent with the Browsertrix-Crawler API/model, so we essentially have two separate crawl engines that we can interact with in consistent ways.

Currently, non-NPLD (by-permission) but domain-crawl frequency URLs (like this one) don't get picked up. It's not clear they get bundled in with the DC, and TBH they probably shouldn't as they won't be marked as BYPM.

So, they should really be treated as annual, and the launcher should be able to launch them without there being a matching current time. The options should also be set up such that they will not force a recrawl (i.e. do not set a launchTimestamp, just use the default recrawl settings, which is a recrawl period of a year.

The primary task is to move the ukwa-manage Luigi task launcher code into here as a standalone command-line tools that can be run manually or invoked from AirFlow.

Then, that tool needs to be updated to use the newer crawl configurations from W3ACT, once they have been added to the crawl feed export, as per ukwa/python-w3act#15

In some situations (I haven't pursued an always reproducible case) a BrokenPipeError is raised, e.g. when piping the report output through head.

This can be "fixed" as per the documentation (h/t Stackoverflow) by catching the error as so:

(eg. in show_raw_stream)

   try:
        for message in consumer:
        :
    except BrokenPipeError:
        # Python flushes standard streams on exit; redirect remaining output
        # to devnull to avoid another BrokenPipeError at shutdown
        devnull = os.open(os.devnull, os.O_WRONLY)
        os.dup2(devnull, sys.stdout.fileno())
        sys.exit(1)  # Python exits with error code 1 on EPIPE

I haven't committed the fix because I don't know how much of an issue it is outside of my environment (it occurs for me using the last example in the readme); I'm not sure that the fix is appropriate; and it makes the code less readable.

Noted here for review in case it's been encountered before and is considered worth implementing.