Comments (6)
Thanks for the heads up, Zane. Just talked to Zach & Katie about how to structure this. Github runner questions outstanding, I'll start on the archiver part of this, following the FERC2 archiver for some inspiration on how to handle year-to-year file structure changes.
from pudl-archiver.
I can have a go at it.
from pudl-archiver.
The EQR may be a bit weird because it's big and also comes in multiple formats. GitHub runners only have 14GB of disk but I think the whole EQR dataset is significantly larger than that (though each individual file won't be).
They don't make the download pages easy to find. The new format (2013-present) uses a kind of annoying javascripty interface but seems to have direct links embedded in it that are formulaic. The older data (pre-2013) splits out transaction tables into a bunch of partitions, and then has the non-trasaction tables for all years in its own separate file.
Right now it looks like each quarterly zipped CSV is around 3GB. So even 1 year might exceed the a GitHub runner's disk space. I wonder how hard / worthwhile it would be to make it possible to serialize the download / upload process from FERC to the runner and then up to Zenodo so the whole dataset isn't ever on disk at the same time.
At least they've finally shut down the janky FTP site and now these can all be downloaded over HTTPS!
from pudl-archiver.
IIRC Zenodo also has a 50GB per deposition limit, but like AWS Open Data it's possible to request a higher quota.
from pudl-archiver.
As a shorter workaround than #44, it should be pretty quick to add a command line flag to restrict EQR to certain partitions - and then we could just split the EQR archival process across like 4 different runners in the actions config.
from pudl-archiver.
I see the archive is on sandbox.zenodo.org. Should it also be available on zenodo.org?
from pudl-archiver.
Related Issues (20)
- Write archiver to regularly run and archive updated CEMS crosswalk HOT 1
- Add validation test that looks at file size HOT 1
- Write individual validation tests as flags in CLI
- Archive FERC EQR pre-2013
- `--dry-run` CLI option is causing failures
- Rogue EIA Thermoelectric Cooling Water Archives? HOT 2
- Archiver workflow sometimes creates unpublished draft depositions HOT 4
- Rewrite EPA CEMS archiver to use API HOT 2
- Specify archiver success & failure conditions HOT 1
- Define `censusdp1tract` archiver success conditions
- Define `eia176` archiver success conditions
- Ensure each XBRL filing in FERC RSS feed has a stable ID HOT 4
- Increase archiver robustness to transient network issues
- Running archiver with `--dry-run` leaves an unpublished draft HOT 2
- Add all new archives to the Catalyst Cooperative community HOT 5
- Update EIA Thermoelectric Cooling Water link matching patterns
- Refresh Zenodo Archives Manually
- Update keywords in new dataset metadata
- "ValueError: I/O operation on closed file" uploading to Zenodo HOT 1
- Archiver no longer works with most recent `pudl@dev` HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pudl-archiver.