Web Crawler
This project provides a web crawler implementation using Scrapy library. It allows crawling a specified website and extracting its content.
- Clone the repository:
git clone https://github.com/your-username/web-crawler.git
- Change into the project directory:
cd web-crawler
- Install the required dependencies:
pip install -r requirements.txt
-
Open the
main.py
file in your preferred text editor. -
Set the desired website URL and domain in the
website
anddomain
variables:
website = 'https://www.example.com/'
domain = 'example.com'
- Run the script:
python main.py
The script will crawl the specified website and print the following information for each page:
- URL: The URL of the page.
- Content: The content extracted from the page.
Contributions are welcome! If you find any issues or want to add new features, please open an issue or submit a pull request.
This project is licensed under the MIT License.