curiouslearner / geeksforgeeksscrapper Goto Github PK

View Code? Open in Web Editor NEW

142.0 10.0 68.0 151 KB

Scrapes g4g and creates PDF

License: MIT License

Python 100.00%

scrapper pdf geeksforgeeks webscraping webscraper hacktoberfest

geeksforgeeksscrapper's Introduction

GeeksForGeeksScrapper

Scrapes GeeksForGeeks and creates html & PDF for chosen category along with syntax highlighting for the code.

Screenshots

Example of articles of C category as html:

Example of articles of C category as pdf:

Installation

To use the scrapper, install the following:

$ sudo apt-get install wkhtmltopdf

Then create venv

$ virtualenv /path/to/g4g-env

Switch to venv

$ source /path/to/g4g-env/bin/activate

Now install BeautifulSoup as:

$ pip install beautifulsoup4

or via package manager as:

$ sudo apt-get install python-bs4

or for Python dependencies, you can just install via requirements.txt inside the virtual environment.

$ pip install -r requirements.txt

Run the G4G_Scrapper

$ python g4g.py

Choose the category you want to scrape from the menu and wait for the magic to happen :)

You can find the output as G4G_<category_name>.html and G4G_<category_name>.pdf in the same directory.

Disclaimer: This is strictly for educational purpose only. Author will not be liable for anything, what so ever; you do with this script.

geeksforgeeksscrapper's People

Contributors

Stargazers

Watchers

geeksforgeeksscrapper's Issues

Why not provide a pdf and html version of all the catergories to download?

I think it will be easy for users.

problem in scrapping algorithms section

Scraping link no: 244 Link: http://www.geeksforgeeks.org/count-number-of-bits-to-be-flipped-to-convert-a-to-b/
Traceback (most recent call last):
File "g4g.py", line 106, in
scrape_category(categoryUrl)
File "g4g.py", line 82, in scrape_category
link_soup = BeautifulSoup(requests.get(link).text)
File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 65, in get
return request('get', url, *_kwargs)
File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 49, in request
response = session.request(method=method, url=url, *_kwargs)
File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 461, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 567, in send
adapter = self.get_adapter(url=request.url)
File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 646, in get_adapter
raise InvalidSchema("No connection adapters were found for '%s'" % url)
requests.exceptions.InvalidSchema: No connection adapters were found for ' http://www.geeksforgeeks.org/count-number-of-bits-to-be-flipped-to-convert-a-to-b/'

Create requirements.txt

Create a requirements.txt file for installing dependencies.

Use of Template Rendering library to seprate the Python and HTML Code

The Current Script to scrape the website contains both python as well as the HTML code . Thus in order to scale the project i.e to modify the HTML code to redesign the Web Page and PDF document, we may need to modify the Script also. Thus it will great to use a template rendering library like Jinja2 to render the HTML content so that the HTML and Python Part Can be separated.