A small project that uses the Scrapy module to scrape a book catalog.
Challenge Instructions: Medium [PT-BR]
- Extracts book data from books.toscrape website to a CSV file;
- Import the data to a Postgres database table;
- Python >=3.9
- Docker-compose
1. Install the dependencies.
python -m pip install -r requirements.txt
2. Initialize the postgres databases.
docker-compose up -d
Just run:
python main.py
- Small study of data types to create the database table.
- Initial learning of the powerful Scrapy library for data scraping.
- I used the python context manager feature in the database class.