data-hunters / tech-skills-visualizer Goto Github PK

StackAPI provides option for passing API keys which increases usage limits. Keys could be passed with ENVs or YAML configuration.

See #5. It should be finished before we start it. Storing data in databases like Neo4j, PostgreSQL/MySQL, HBase, etc. could be beneficial.

We should store collected data in files for at least three reasons:

1. Plotting graph again without crawling.
2. Using gathered data for other purposes.
3. Retrying the whole process when crawling is stopped (e.g. IP/account reached threshold).

Goal of this feature is to implement only storing functionality. We should use some popular format like JSON or/and CSV.

Please change simple dict objects to TypeDict in crawler and transformer. It will make more clear and manageable the whole pipeline.