Giter VIP home page Giter VIP logo

eleme-crawler's Introduction

饿了么商家数据爬虫(多线程版本在multiper文件里)

###########环境依赖

        python3
        所需要的支持库:mysql_connector、requests
        Mysql

###########操作步骤

首先:在run.py同级文件夹里新建一个cookies.txt的空文件
首先:在run.py同级文件夹里新建一个cookies.txt的空文件
首先:在run.py同级文件夹里新建一个cookies.txt的空文件

一、配置数据库:
(本爬虫采用MySQL数据库)
进入eleme_mysql.py,修改数据库连接、密码、端口,数据库名、数据表名(需要提前建立含有字段的数据库)

二、配置地理位置:
在run.py中找到如下代码:
' def __init__(self):
    self.latitude = 22.52680
    self.longitude = 113.93082'
 通过谷歌地图找到你所需要位置的经纬度,替换上面的经纬度即可

三、修改爬取页数:
在run.py中找到如下代码:
'#修改爬取页数(乘以8)
    maxpage=31
    for i in range(0, int(maxpage)):
        page_num = i * 8
        self.main_parse(page_num)'
        
四、爬取:
运行run.py开始抓取商家数据(只包含商家名字、商品名字和商品月销量,需要添加其他字段请参考json文件自行修改)
(第一次运行会要求填入手机号获取验证码,之后cookies保存在cookies.txt文件中)

###########声明

本程序作者:GallenQiu

仅供个人学习,禁止用于商业用途

备注:本爬虫抓取端口是手机H5页面,大量连续访问会被封号一段时间。

eleme-crawler's People

Contributors

gallenqiu avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar

eleme-crawler's Issues

获取validate_token时报错

第一次登录调用send的时候会返还这个:'{"message":"账户存在风险,需要滑动验证码","name":"NEED_SLIDE"}',请问怎么解决呢

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.