Archive FERC EQR about pudl-archiver HOT 6 CLOSED

Thanks for the heads up, Zane. Just talked to Zach & Katie about how to structure this. Github runner questions outstanding, I'll start on the archiver part of this, following the FERC2 archiver for some inspiration on how to handle year-to-year file structure changes.

from pudl-archiver.

I can have a go at it.

from pudl-archiver.

The EQR may be a bit weird because it's big and also comes in multiple formats. GitHub runners only have 14GB of disk but I think the whole EQR dataset is significantly larger than that (though each individual file won't be).

They don't make the download pages easy to find. The new format (2013-present) uses a kind of annoying javascripty interface but seems to have direct links embedded in it that are formulaic. The older data (pre-2013) splits out transaction tables into a bunch of partitions, and then has the non-trasaction tables for all years in its own separate file.

Right now it looks like each quarterly zipped CSV is around 3GB. So even 1 year might exceed the a GitHub runner's disk space. I wonder how hard / worthwhile it would be to make it possible to serialize the download / upload process from FERC to the runner and then up to Zenodo so the whole dataset isn't ever on disk at the same time.

At least they've finally shut down the janky FTP site and now these can all be downloaded over HTTPS!

from pudl-archiver.

IIRC Zenodo also has a 50GB per deposition limit, but like AWS Open Data it's possible to request a higher quota.

from pudl-archiver.

As a shorter workaround than #44, it should be pretty quick to add a command line flag to restrict EQR to certain partitions - and then we could just split the EQR archival process across like 4 different runners in the actions config.

from pudl-archiver.

I see the archive is on sandbox.zenodo.org. Should it also be available on zenodo.org?

from pudl-archiver.