• It will be a continuation of your Second task.
• Refer to the second shortlisting task, instead of CSV store those scraped data in a MySQL or PostgreSQL Database.
• Here is a video of how to use MySQL with Python: https://www.youtube.com/watch?v=3vsC05rxZ8c
• Make a short video explaining and running the task in <10mins. Don't need to show long-running codes.
• Upload your code to a GitHub public repository
• Submit the video and the GitHub link for the same in the submission form

1. Set up config.ini as per requirements for the MySQL database.
2. Simply install the required libraries.

pip install -r requirements.txt

Maintaining a virtual environment is recommended.

To run the code simply run the following.

python main.py

Once the output is ready, it will be stored in a database called twitterbase in table users. You can use list.py to list the entries.

Bonus: Multiprocessing. Even with MySQL! Scraping runs on multiple processes. This creates a much more scalable result.

1. Analysis of required data.
2. Building a basic pipeline for scraping.
3. Testing if the current scraping pipeline works (see test.ipynb).
4. Application of the pipeline (see scraper.py).
5. Deployment.