One can either run pip install -r requirements.txt
to install all the required dependencies, or use setup.py
to do so (through calling python setup.py build
or python setup.py install
)
Under the crawler
folder, crawler.py
stores the function definitions and generic_testing.py
contains the generic testing codes (without unit tests yet)
tests
folder contains files that will be used to run unit tests in the future (currently not implemented), if they are needed later.