Giter VIP home page Giter VIP logo

Comments (3)

braindevices avatar braindevices commented on May 24, 2024

my ip is not blocked by any of those search engines. I try to debug a little bit by output the url and html with following code:

from search_engine_parser.core.engines.google import Search as GoogleSearch

gsearch = GoogleSearch()
search_args = ('python google search api', 1)
gsearch.search(*search_args, cache=False, hl="en")

The url is https://www.google.com/search?start=0&q=python+google+search+api&gbv=1&hl=en
And I found the resulting html from get_source() saying:

Before you continue to Google Search
Google uses cookies and data to:

    Deliver and maintain services, like tracking outages and protecting against spam, fraud, and abuse
    Measure audience engagement and site statistics to understand how our services are used

If you agree, we’ll also use cookies and data to:

    Improve the quality of our services and develop new ones
    Deliver and measure the effectiveness of ads
    Show personalized content, depending on your settings
    Show personalized or generic ads, depending on your settings, on Google and across the web

For non-personalized content and ads, what you see may be influenced by things like the content you’re currently viewing and your location (ad serving is based on general location). Personalized content and ads can be based on those things and your activity like Google searches and videos you watch on YouTube. Personalized content and ads include things like more relevant results and recommendations, a customized YouTube homepage, and ads that are tailored to your interests.

Click “Customize” to review options, including controls to reject the use of cookies for personalization and information about browser-level controls to reject some or all cookies for other uses. You can also visit g.co/privacytools anytime.

So it seems like you need to confirm something and save some cookie before you could go on.

from search-engine-parser.

braindevices avatar braindevices commented on May 24, 2024

bingsearch only works once per 5 to 8 trials. no idea why though.

from search-engine-parser.

neithere avatar neithere commented on May 24, 2024

Looks like the consent cookie should be set, e.g. like this: https://stackoverflow.com/a/70560365/68097

Unfortunately the lib doesn't seem to be configurable in this aspect. The cookies argument should be passed to aiohttp.ClientSession() in CacheHandler.get_source(): https://github.com/bisohns/search-engine-parser/blob/master/search_engine_parser/core/utils.py#L78

FYI, I've added a very crude workaround here: a537f4e
Obviously a proper implementation should add an engine-dependent set of default cookies or something in this vein.

from search-engine-parser.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.