bishwas-py / cruel Goto Github PK

Cruel is a Python web-request library that allows you to extract detailed information from any webpage or site.

• Inbuilt beautiful soup instance
• ScraperAPI powered; so you don't have to worry about IP blocking

You can install the cruel using pip:

pip install cruel

Below are examples of how to use the cruel to extract data from Fiverr gig pages and user profiles.

from cruel import session

session.set_scraper_api_key("XYZ-SCRAPER_API_KEY")
response = session.get("https://www.fiverr.com/username/your-gig-slug")  # your fiverr url should be here
print(response.soup)  # gives you beautiful soup instance
# You can use `response.soup` to further extract your information. 

Get your ScraperAPI key here.

The cruel is organized into several modules to enhance code readability and maintainability:

• cruel
  • __init__.py: For exporting session
• cruel.utils
  • req.py: Extending requests for Fiverr scraping
  • scrape_utils.py: Utilities for scraping

GPL

New pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.

Check more of my projects.