chenjiandongx / 51job-spider Goto Github PK

View Code? Open in Web Editor NEW

403.0 403.0 199.0 10.3 MB

🔎 前程无忧 Python 招聘岗位信息爬取和分析

License: MIT License

Python 100.00%

51job python spider

51job-spider's People

Contributors

Stargazers

Watchers

Forkers

mercynoo 957204459 buyuhuang mazelinux awesome-archive lujc lovecn flysky1991 danielchem guiguzi1110 cuihubin2018 wxjlotus yikedouer modi1492 lining0806 fashtimedotcom lynnljl fanlongxing kasaimiluo wang935415150 tangshenglong chinesehuang henfee tonyjona johnnyzhao801 luckypython rubinliudongpo helloworldtang lhqs amwihtfei ruijee 521314 rebortboss awinterbunny hidesoon zlszhonglongshen codinglv jaapteam huahuijay angelife tammy0825 youthhn diego2048 arsenal0927 damugeisme gavingaogao lishuanwang jiaqi0109 yunpengzhao ckzdy ysj123688 tongyuanfeng yiweishen itjinkui purityzw xclu soongxueyong yuzhen233 lnming hengthu coolsnake wocalei bestdarkhorse yuzihao666 paulluo alige32 jiebai720 restoflife1997 jiguang123 githubdaxueseng eddyzheng goodluckwlx wrp142536 welaing geyixin chenyuan2048 lishulincug duchencheng huberyzhou wqh0109663 hehuanshu96 jinliugithub chenzanlee dahaoniuniu hyj1116 jk50505k wuhao111 zhaozhen2018 vickzhang songcao emaiqi know1ng netgb suiyuchen lishu1125 tuzi5156 sdpku melody5417 pengjason awesomesunny

51job-spider's Issues

we can see lot "???" in post_desc_counter.csv

Will you please give me a format of this file? currently it is not "UTF-8".

beautifulsoup解析网页：'NoneType' object has no attribute 'text'

该url模板第一页的第一条数据的网页结构和其他都不一样，导致解析错误。解析的时候try过滤或者单独加一个逻辑来处理。

2019-09-23 01:04:32,404 - 爬取第 748 条岗位详情
爬完之后就一直停滞在这了，只出现post_require_new.txt不太懂哪里出了出了差错
中间有一段儿是:
Traceback (most recent call last):
File "src\gevent\greenlet.py", line 766, in gevent._greenlet.Greenlet.run
File "C:/Users/Administrator/Job/job_spider.py", line 106, in post_require
html = resp.content.decode("gbk")
UnicodeDecodeError: 'gbk' codec can't decode byte 0xae in position 1060: illegal multibyte sequence
2019-09-22T17:03:11Z <Greenlet at 0x178d629c158: <bound method JobSpider.post_require of <main.JobSpider object at 0x00000178D6288C88>>> failed with UnicodeDecodeError

您能帮忙解惑吗？

薪资计算错误

x + (y - x) * 0.4 应该改为 (x + y) * 0.5 * 0.5，前一个0.5是取区间中值，后一个0.5是除去虚假招聘浮沫(乐观估计)

大神，能更新下代码吗，现在不能用了呀

href, post = b.find("a")["href"], b.find("a")["title"]
这行代码会报错了

有个地方是不是错了

counter[seg]` = counter.get(seg, 1) + 1

默认seg这个地方是不是应该是0，因为你后面还有一个加1，你想如果没有找到seq的话，你把默认值搞成1，后面你又加一个1，那这个第一次出现的dict value不就是2了？

counter[seg]` = counter.get(seg, 0) + 1

是不是应该是上面的这样

有兴趣来阿里云工作吗？请加我微信cn-qiyu

hey guy I got a good idea why not try to deploy it to a website ?

爬100页左右的数据后就被封了

我改了改你的代码，想多爬点数据，结果大致爬了100页左右以后就被封了。之后就再也获取不到数据了，而且....开代理，改IP都，改浏览器标识都不行，51job反爬虫这么牛逼啊

完美的大职作业！

AttributeError: 'NoneType' object has no attribute 'find_all'

Traceback (most recent call last):
File "job_spider.py", line 338, in
spider.run()
File "job_spider.py", line 329, in run
self.job_spider()
File "job_spider.py", line 90, in job_spider
bs = BeautifulSoup(html, "lxml").find("div", class_="dw_table").find_all("div", class_="el")
AttributeError: 'NoneType' object has no attribute 'find_all'

反爬怎么处理的呀

你好,作者

请问你的可视化制图工具是什么,图表制作的很好看