This application can be used as either a web api or a standalone script.
- Node.js
- Reids running in a docker container
- Provide values for config file before running
npm start
node script
- Websockets should be implemented to allow for long running task via the web api
- Provide email notification of comepleted tasks
- Accept custom URL rule sets as argument when crawler starts