The scraper is used to scrape the study programs and related courses from the Faculty of Computer Science and Engineering at the Ss. Cyril and Methodius University in Skopje. which can be found at the following url.
The scraper will save the scraped data in three different files:
study_programs.csv
: contains the details of the study programscurriculum.csv
: contains the details of the study programs and related coursescourses.csv
: contains the details of the courses
- Python 3.9 or later
Before running the scraper, make sure to set the following environment variables:
OUTPUT_DIRECTORY_PATH
: the path to the directory where the output files will be savedMAX_WORKERS
: the maximum number of workers that will be used to scrape the websiteEXECUTOR_TYPE
: the type of executor that will be used to scrape the website. The possible values are "THREAD" and "PROCESS"
-
Clone the repository
git clone <repository_url>
-
Install the required packages
pip install -r requirements.txt
-
Run the scraper
python main.py
Make sure to replace <repository_url>
with the actual URL of the repository.