collect and organize data into a T1 data lake and T2 tables. named after the Aswan Dam

will probably need to separate a few things from it:

• t2extractor
  • unstructured json to tabular data automatically
  • aswan.t2.extractor
• scheduler

TODO

• dvc integration
• export to dataset template
  • maybe part of the dataset
• cleanup requirements
• s3, scp for push/pull
• add verified invalid output that is not parsing error
• selective push / pull
  • with possible nuking of remote archive
  • cleaning local obj store (when envs blow up, ide dies)
• parsing/connection error confusion
  • also broken session thing
• conn session cpu requirement
• resource limits
• transfering / ignoring cookies
• lots of things with extractors
• template projects
  • oddsportal
    • updating thingy, based on latest match in season
  • footy
  • rotten
  • boxoffice