Toy URL web crawler done for a take-home assignment.
You need to have Python 3.8+ and poetry installed. Afterwards run:
$ poetry installTo test if the installation worked, you can use:
$ poetry run web-crawler -h
usage: Web Crawler [-h] -u URL [-d DEPTH] [-w WORKERS] [-j] [--debug] [--version]
A simple Python web crawler
optional arguments:
-h, --help show this help message and exit
-u URL, --url URL URL to crawl
-d DEPTH, --depth DEPTH
depth limit (default: 2)
-w WORKERS, --workers WORKERS
maximum number of workers (default: 10)
-j, --json output JSON instead of Python's internal representation
--debug enable debug log (WARNING: very verbose)
--version show program's version number and exitTo run the tests:
$ poetry run pytest -vvYou can also run black to check for code formatting and mypy to check for types:
$ poetry run pytest --black --mypy --mypy-ignore-missing-importsThere is also a Makefile available describing some common commands.
Another option to install is to use the provided Dockerfile:
$ docker build -t web-crawler .
$ docker run -it --rm web-crawler pytest -vv
$ docker run -it --rm web-crawler web-crawler --helpCrawling a page (the protocol is required!):
$ poetry run web-crawler -u 'https://google.com'By default this will recursively crawl the page until hitting the limit of 2.
You can increase this limit by using -d parameter:
$ poetry run web-crawler -u 'https://google.com' -d 4To not overload the server with requests, the maximum number of simultaneous
connections is limited. By default up to 10 workers will work concurrently, if
you want to change this limit use the -w parameter:
$ poetry run web-crawler -u 'https://google.com' -w 20If you want to output using JSON representation instead of Python's internal
one, you can pass the -j flag to it:
$ poetry run web-crawler -u 'https://google.com' -jTip: You can filter the JSON output using jq:
$ poetry run web-crawler -u 'https://google.com' -j | jq '."https://google.com"."https://google.com/services"'If you want to debug the application there is also a debug log (warning: very verbose):
$ poetry run web-crawler -u 'https://google.com' -j --debugSee Architecture.md file.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent