Built using Python and Beautiful Soup. Parses University academic sites and compiles papers, articles, and other academic material into a .csv file.
I've used Visual Studio to create my virtual environment:
- Open the solution.
- Right click on 'Python Environments'.
- Click 'Add Environment'.
- Make sure the new environment name is 'venv'.
- Create a new virtual environment based on requirements.txt.
This can also be done via the command line, however, I had some trouble integrating the environment with Visual Studio when creating it this way. If you're not using Visual Studio then this method will work.
cd UniversityWebScraper
python -m venv ./venv
"venv/Scripts/activate"
pip install -r requirements.txt
An executable file has been included for ease of use. Open a command prompt and cd to the root of the cloned/downloaded repository. Then run this command to get a list of command line arguments:
PyUniScraper --help
You can use this executable to get .csv files that contain academic material information for your chosen university.