Distributed, continuous image crawler.

Firefox web browser (gecko driver)

Creating a virtual environment for python recommended.

python3 -m venv ./venv

Then install dependent packages.

pip3 install -r requirements.txt

In order to keep the monitoring running, a monitoring server must be set up before crawling node starts. Start running the monitor server using this command:

python3 cherryServer.py

Start crawling using the following command.

python3 crawler.py --[option] [option_value]

Avialable options are:

• --site [site] target site to crawl (instagram, facebook, etc.)
• --filter [filter_type] type of data filter to screen the data (face)
• --nthread [number_of_threads] number of threads used to load web driver and start crawling
• --logpath [folder_name] folder name to save the logs in

The status of crawling may be monitored using the monitor reader.

python3 monitor_read.py