If you want API data copied to your database, the workers should fetch for you elastically.
use schema.sql to create a database, and load it with the types of information you need
see samples/usage.py for an example. Create an api method and save method for your workflow by subclassing BaseWorker.
- Splits update task into pieces
- Prioritizes pieces
- Can be paused and resumed
- Can scale across hosts easily using DB
- Good test coverage (soon :) )
See Sample folder