collect and organize data into a T1 data lake and T2 tables. named after the Aswan Dam
will probably need to separate a few things from it:
- t2extractor
- unstructured json to tabular data automatically
- aswan.t2.extractor
- scheduler
TODO
- dvc integration
- export to dataset template
- maybe part of the dataset
- cleanup requirements
- s3, scp for push/pull
- add verified invalid output that is not parsing error
- selective push / pull
- with possible nuking of remote archive
- cleaning local obj store (when envs blow up, ide dies)
- parsing/connection error confusion
- also broken session thing
- conn session cpu requirement
- resource limits
- transfering / ignoring cookies
- lots of things with extractors
- template projects
- oddsportal
- updating thingy, based on latest match in season
- footy
- rotten
- boxoffice
- oddsportal