- It will be a continuation of your Second task.
- Refer to the second shortlisting task, instead of CSV store those scraped data in a MySQL or PostgreSQL Database.
- Here is a video of how to use MySQL with Python: https://www.youtube.com/watch?v=3vsC05rxZ8c
- Make a short video explaining and running the task in <10mins. Don't need to show long-running codes.
- Upload your code to a GitHub public repository
- Submit the video and the GitHub link for the same in the submission form
- Set up
config.ini
as per requirements for the MySQL database. - Simply install the required libraries.
pip install -r requirements.txt
Maintaining a virtual environment is recommended.
To run the code simply run the following.
python main.py
Once the output is ready, it will be stored in a database called twitterbase
in table users
.
You can use list.py to list the entries.
Bonus: Multiprocessing. Even with MySQL! Scraping runs on multiple processes. This creates a much more scalable result.
- Analysis of required data.
- Building a basic pipeline for scraping.
- Testing if the current scraping pipeline works (see test.ipynb).
- Application of the pipeline (see scraper.py).
- Deployment.