meibenjin / googlesearchcrawler Goto Github PK

View Code? Open in Web Editor NEW

385.0 35.0 256.0 169 KB

a tool for crawl Google search results

License: MIT License

Python 100.00%

googlesearchcrawler's Introduction

Google Search Crawler

This is a simple google search results crawler. Before use this tool, please read these tips below.

Requirements

You may find requirements in requirements.txt which can be installed using pip install -r requirements.txt

More info:

Python

python should be installed in your computer. here is the official website: http://www.python.org
BeautifulSoup

A html parser to extract search results from Google. BeautifulSoup(version 4) is better.

For more information about BeautifuleSoup, please visit: http://www.crummy.com/software/BeautifulSoup/
dotenv (python-dotenv)

How to Use?

Rename or copy .env.example into .env. Apply your own config in this file only.
single key word

python gsearch.py 'your query key words'

It will return about 10 extracted results by default. if you need more results, just change the expect_num value.
list of key words

python gsearch.py

First create a file named keywords, put your key words list into this file, one key word per line.

If there are any problems or bugs about this tool, please open an issue.

googlesearchcrawler's People

Contributors

Stargazers

Watchers

Forkers

qqshow caimaoy eagles2f jonney sdhelo sdlearn snail200x snowody litsand kerneltravel richardlxc hzhaoaf xuyuandong marco90 successor hewaygithub tianyujia iam007site wukoo wanghaisheng aidan-huang lqniunjunlper zimndev ericjxl2013 ryeyao lantares jlinleung majianbing cnlouts morningmoni ifff mnjstwins dawnminghuang woodwardoge tylerzhangzc kvdaniel yushen777 52nlp moonbay caigaopei mjbeverley shdut praveenjha527 4sp1r3 edmondztt wesavetheworld leveryd shuaigelee dotmark foogle sugartom jefferychenpku zhangzhensong pythonone focus3d ojlee yusa95 walkinreeds fosterwei cbccb8 misspast parasuc lqruui supertanglang maizifang hsd315 bahaahassanieh veterun yongxu74 dizheng barrycug thientu alancpu 0x24bin hi-cat thornbird91 marzlia andywlh joncv uj00007 fl2lee ann-qin-lu jijicanyu abhashkumar fpcmotif daisyzhouch py-karl infinityfeng lihongqiang teserak 9012wushuang cs24 whxloveyrh test123test111 cheaphunter lycansfree forhunter secpersu housoso jerifer

googlesearchcrawler's Issues

无法运行

/usr/local/lib/python2.7/site-packages/bs4/init.py:181: UserWarning: No parser was explicitly specified, so I'm using the best available HTML parser for this system ("lxml"). This usually isn't a problem, but if you run this code on another system, or in a different virtual environment, it may use a different parser and behave differently.

The code that caused this warning is on line 215 of the file gsearch.py. To get rid of this warning, change code that looks like this:

BeautifulSoup([your markup])

to this:

BeautifulSoup([your markup], "lxml")

markup_type=markup_type))

Is it possible to iterate through the various results pages? (to get more than 100 results)

Hi,

Thanks for sharing this.

I wonder if it's possible to make it iterate through more results pages on a single keyword search, to get more search results (more than 100)

There is a mistake in line 105

In line 105, there should be changed to 'div' instead of 'li'

Doesn't print unless I remove reload(sys) sys.setdefaultencoding('utf-8')

I had to comment out the following lines to get any output or print statements to work.

reload(sys)

sys.setdefaultencoding('utf-8')

Save to database?

Hi, this script can save result to database?

Suggesting TODO list

I have this todo list for the beginning. if there is no objection, I work on it.

make config file for settings like google url, no. of result,...
clean up the result by removing <b> tags
make the result on running directly from terminal, more terminal friendly (ex. replace => with :)
make the code usable as a module

I think extractSearchResults(html) might be not valid now

It returns an empty list

how to solve this error

File "gsearch.py", line 35
print "ERROR: Make sure you have .env file with proper config"
^
SyntaxError: Missing parentheses in call to 'print'. Did you mean print("ERROR: Make sure you have .env file with proper config")?

求助

您好，请问我在爬取的时候，代理服务器正常，一直捕获到这个异常，求解
hi,when I crawled, I have been caught this exception,plz!

error: 'module' object is not callable

ub16hp@UB16HP:/media/ub16hp/WINDOWS/ub16_prj/GoogleSearchCrawler$ python3 gsearch.py
error: 'module' object is not callable

error: 'module' object is not callable
error: 'module' object is not callable

Can you please adjust to python3?

First of all, thanks a lot.
I got the following error
error: initial_value must be str or None, not bytes
after adjusting the code to python3 and I'm not sure what is the issue.

I have attached the changes I did.

gsearch.zip