Rather than perform the main copy in Luigi, we could run Apache Flume when pushing files up to HDFS. It's an extremely well-established and mature tool that's built to do exactly this.

We could use a Spooling Directory Source for *.warc.gz files, and a Taildir Source to continuously process crawl log files. These would be sent to a suitable HDFS Sink.

The directory sources should be set to not delete files (see deletePolicy), and to put the file path in the fileHeader so we can use ${file} to reconstruct the path on HDFS. (see rough outline here)

A second Luigi-powered process would check for files and delete them when the checksums of both had been verified.

Should be pretty easy to Dockerize, and use a Java 8 base image rather than something like this.

Modern versions of HDFS support an iNotify API. Here's an example: https://github.com/onefoursix/hdfs-inotify-example

We could wrap this daemon up as e.g. a Storm Spout (although this is being considered already and there are issues around scaling). Alternatively, we could just run it as a plain daemon that publishes to e.g. Kafka.

Note that the overall design would have to assume this would sometimes miss things, and make it possible to (re-)run and processing via occasional hadoop fs -lsr jobs.