Giter VIP home page Giter VIP logo

cnblogs_spider's Introduction

cnblogs_spider

博客园爬虫_基于Nodejs+mongodb

个人练习（欢迎交流），代码内有详细备注

功能

  • 抓取博主信息及对应的所有文章信息
  • 存入数据库，分析
  • 简单修改代码可以做到更深层抓取...可自行修改哈，此项例子只做参考

逻辑

  1. 以首页为入口，抓取4000条初始文章url，匹配数据库剔除重复
  2. 解析文章页url，获取用户主页url
  3. 解析用户主页，判断是否存在多个页面，获取当前用户所有文章url，匹配数据库剔除重复
  4. 抓取文章，调用数据库更新用户数据及博客数据

使用说明及启动

第一步：安装依赖 ————npm install 
第二步：运行mongodb ————mongod --dbpath c:\mongo 
第三步：运行爬虫 ————node app.js

欢迎一同学习交流，且学且努力

蒋启钲 [email protected]

cnblogs_spider's People

Contributors

jiangqizheng avatar

Watchers

Henry Kesson avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.