Giter VIP home page Giter VIP logo

scrapyd_web_log's Introduction

scrapy + scrapyd + scrapydweb + logparser 分布式部署

1.构建 scrapyd_logparser
cd scrapyd_logparser
docker build -t scrapyd_logparser .
2.运行 scrapyd_logparser
docker run -d -p 6800:6800 --name scrapyd_1 scrapyd_logparser
# docker run -d -p 6800:6800 -v /root/scrapyd_logparser:/code --name scrapyd_1 scrapyd_logparser
# 可以外联出文件 可以进行配置修改
3.构建 scrapydweb
cd scrapydweb
docker build -t scrapydweb .
4.运行 scrapydweb
docker run -d -p 5000:5000 -v /root/scrapydweb:/code --name scrapydweb scrapydweb
# 外链出文件 好修改配置 进行更新部署
# 修改配置
vim scrapydweb_settings_v10.py 
# 重启
docker restart scrapydweb
5.多机部署
在其他机器中 部署 文件 执行 1,2 步
同时在 scrapydweb 的配置文件 添加在 新主机的ip

# 以下添加
SCRAPYD_SERVERS = [
    '192.168.5.131:6800',
	'192.168.5.131:6805',
    # 'username:password@localhost:6801#group',
    # ('username', 'password', 'localhost', '6801', 'group'),
]

: 可以自己查看 一下 scrapydweb 配置文件 修改,和 scrapyd 配置文件修改

参考:

  1. scrapydweb
  2. logparser

scrapyd_web_log's People

Contributors

dependabot[bot] avatar libaibuaidufu avatar

Watchers

 avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.