You can use pip to install the requirements as follows:

pip install -r requirements.txt

You only need to type

python3 crawl3.py your-start-url timeout researchlevel

in your terminal. And then input the starturl ,timeout and researchlevel. The result is saved in the file named "data.json" in the same folder as well as STDOUT.

The research level can be set -1 if you want to explore the whole internet

You can run the test case by following the instruction:

1 python3 mockserver.py to run the mockserver for the test case
2 python3 -m unittest unit_test pagecrawlbehaviour_test

docker build -t crawl
docker run crawl crawl3.py your-start-url timeout researchlevel