Comments (1)
A workaround of using pagination()
with parse_qsl
, urlsplit
from urllib.parse
:
from serpapi import GoogleSearch
from urllib.parse import (parse_qsl, urlsplit)
params = {
'api_key': '...', # serpapi api key
'engine': 'google', # search engine
'q': 'minecraft', # search query
'hl': 'en' # language
# 'start': 0 # or explicitly add page number
}
search = GoogleSearch(params)
# iterate over all pages
while True:
results = search.get_dict()
print(f"Currenlty on page: {params.get('start')}")
if 'error' in results:
print(results['error'])
break
# for result in results.get('organic_results', []):
# print(result.get('position'), result.get('title'), sep='\n')
# check if the next page key is present in the JSON
# if present -> split URL in parts and update to the next page
if 'next' in results.get('serpapi_pagination', {}):
search.params_dict.update(dict(parse_qsl(urlsplit(results.get('serpapi_pagination').get('next')).query)))
else:
break
Output:
Currenlty on page: None # first page. if 'start' param is present and set explicitly to 0, None will become 0
Currenlty on page: 10 # second page
Currenlty on page: 20
Currenlty on page: 30
Currenlty on page: 40
Currenlty on page: 50
Currenlty on page: 60
Currenlty on page: 70
Example from the dashboard -> your searches page (Inspect last request (8th page), next
hash key is empty thus exists pagination) :
from google-search-results-python.
Related Issues (20)
- How to get "related articles" links from google scholar via serpapi? HOT 3
- [Version] Update PyPi to include the most up-to-date version HOT 3
- Setting Timeout on Instantiation for GoogleSearch HOT 3
- .. HOT 2
- AttributeError during the import of the module: `initialized module 'serpapi' has no attribute 'GoogleSearch' (most likely due to a circular import)` HOT 5
- Exception not handled on SerpApiClient.get_json HOT 1
- [Feature Request] Add Async Implementation HOT 5
- [Discuss] Wrapper longer response times caused by some overhead/additional processing
- [Feature request] Make `async: True` do everything under the hood HOT 1
- How to add proxy and ssl certs while using the Search API? I am currently trying to access it via my corporate network. HOT 1
- requests.exceptions.SSLError: HTTPSConnectionPool(host='serpapi.com', port=443) HOT 1
- Python package should not include tests
- SSLCertVerificationError [SSL: CERTIFICATE_VERIFY_FAILED] error HOT 1
- Python 3.8+, Fatal Python error: Segmentation fault when calling requests.get(URL, params) with docker python-3.8.2-slim-buster/openssl 1.1.1d and python-3.9.10-slim-buster/openssl 1.1.1d HOT 1
- ImportError: cannot import name 'GoogleSearch' from 'serpapi' HOT 9
- Google lens can't find my image
- ImportError: cannot import name 'GoogleSearch' from 'serpapi' python 3.10 HOT 2
- get_html() Returns JSON Instead of HTML
- Different results from serpapi (Google Trends) versus Google Trends site HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from google-search-results-python.