Topic: spider Goto Github

👇 Here are 2694 public repositories matching this topic...

201206030 / novel-plus
spider,novel-plus 是一个多端（PC、WAP）阅读 、功能完善的小说 CMS 系统。包括小说推荐、小说检索、小说排行、小说阅读、小说书架、小说评论、小说爬虫、会员中心、作家专区、充值订阅、新闻发布等功能。
User: 201206030
Home Page: https://novel.xxyopen.com
novel crawl read book spider

5ime / video_spider
spider,短视频去水印：抖音,皮皮虾,火山,微视,微博,绿洲,最右,轻视频,快手,全民小视频,巴塞电影,陌陌,Before避风,开眼,Vue Vlog 小咖秀,皮皮搞笑,全民K歌,西瓜视频,逗拍,虎牙,6间房,梨视频,新片场,acfun,美拍...
User: 5ime
Home Page: https://lab.5ime.cn/video
spider video php

andeya / pholcus
spider,Pholcus is a distributed high-concurrency crawler software written in pure golang
User: andeya
spider crowler

bda-research / node-crawler
spider,Web Crawler/Spider for NodeJS + server-side jQuery ;-)
Organization: bda-research
crawler javascript spider extract-data cheerio jquery nodejs

boris-code / feapder
spider,🚀🚀🚀feapder is an easy to use, powerful crawler framework | feapder是一款上手简单，功能强大的Python爬虫框架。内置AirSpider、Spider、TaskSpider、BatchSpider四种爬虫解决不同场景的需求。且支持断点续爬、监控报警、浏览器渲染、海量数据去重等功能。更有功能强大的爬虫管理系统feaplat为其提供方便的部署及调度
User: boris-code
Home Page: http://feapder.com
scrapy feapder spider crawler python feaplat

brucedone / awesome-crawler
spider,A collection of awesome web crawler,spider in different languages
User: brucedone
web-crawler crawler web-scraper spider node-crawler scraper awesome

charlespikachu / decryptlogin
spider,DecryptLogin: APIs for loginning some websites by using requests.
User: charlespikachu
Home Page: https://httpsgithubcomcharlespikachudecryptlogin.readthedocs.io/zh/latest/
requests login python3 spider crawler zhihu bilibili weibo taobao jingdong

crawlab-team / crawlab
spider,Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架
Organization: crawlab-team
Home Page: https://www.crawlab.cn
webcrawler scrapy crawlab spiders-management go scrapyd-ui spider crawler webspider web-crawler

dedsecinside / torbot
spider,Dark Web OSINT Tool
Organization: dedsecinside
tor tor-network deepweb dark-web psnappz crawler spider dedsec-inside python-web-crawler security-tools

dormymo / spiderkeeper
spider,admin ui for scrapy/open source scrapinghub
User: dormymo
Home Page: http://sk.7mdm.com:5000/
scrapy dashboard scrapyd scrapy-ui scrapyd-dashboard scrapyd-ui spider

elliotgao2 / gain
spider,Web crawling framework based on asyncio.
User: elliotgao2
python crawler spider asyncio uvloop aiohttp

elliotgao2 / toapi
spider,Every web site provides APIs.
User: elliotgao2
Home Page: https://gaojiuli.github.io/toapi/
api crawler flask html json python spider toapi web

evil0ctal / douyin_tiktok_download_api
spider,🚀「Douyin_TikTok_Download_API」是一个开箱即用的高性能异步抖音、快手、TikTok、Bilibili数据爬取工具，支持API调用，在线批量解析及下载。
User: evil0ctal
Home Page: https://douyin.wtf
python pywebio tiktok douyin api scraper fastapi no-watermark online-parsing async

facert / awesome-spider
spider,爬虫集合
User: facert
spider python awesome

gerapy / gerapy
spider,Distributed Crawler Management Framework Based on Scrapy, Scrapyd, Django and Vue.js
Organization: gerapy
Home Page: https://docs.gerapy.com/
scrapy distributed webspider scrapyd dashboard spider django vuejs gerapy vue docker

geziyor / geziyor
spider,Geziyor, blazing fast web crawling & scraping framework for Go. Supports JS rendering.
Organization: geziyor
crawler go scraper scraping spider

gocolly / colly
spider,Elegant Scraper and Crawler Framework for Golang
Organization: gocolly
Home Page: https://go-colly.org/
golang scraper framework crawler scraping crawling spider go

guyueyingmu / avbook
spider,AV 电影管理系统， avmoo , javbus , javlibrary 爬虫，线上 AV 影片图书馆，AV 磁力链接数据库，Japanese Adult Video Library,Adult Video Magnet Links - Japanese Adult Video Database
User: guyueyingmu
javbus avmoo javlibrary spider crawler laravel scraper adult magnet-link magnet

howie6879 / owllook
spider,owllook-小说搜索引擎
User: howie6879
Home Page: https://www.owllook.net
spider python crawler python3 ruia novel book asyncio asyncio-spider sanic

howie6879 / ruia
spider,Async Python 3.6+ web scraping micro-framework based on asyncio
User: howie6879
Home Page: https://www.howie6879.com/ruia/
asyncio aiohttp asyncio-spider crawler crawling-framework spider uvloop ruia python-ruia middlewares

hu17889 / go_spider
spider,[爬虫框架 (golang)] An awesome Go concurrent Crawler(spider) framework. The crawler is flexible and modular. It can be expanded to an Individualized crawler easily or you can use the default crawl components only.
User: hu17889
spider crawler go schedule pipeline

ihmily / douyinliverecorder
spider,可循环值守和多人录制的直播录制软件，支持抖音、TikTok、快手、虎牙、斗鱼、B站、小红书、pandatv、afreecatv、flextv、popkontv、twitcasting、winktv、百度、微博、酷狗、花椒、流星、Twitch等平台直播录制
User: ihmily
douyin-live video-downloader douyin-api douyulive live-recorder python tiktok-api tiktoklive spider afreecatv

jae-jae / querylist
spider,:spider: The progressive PHP crawler framework! 优雅的渐进式PHP采集框架。
User: jae-jae
Home Page: https://querylist.cc
querylist crawler spider scraper

jaeles-project / gospider
spider,Gospider - Fast web spider written in Go
Organization: jaeles-project
spider crawler go gospider bugbounty

javclub / core
spider,🔞 JAVClub - 让你的大姐姐不再走丢
Organization: javclub
jav javbus adult adult-content video-streaming google-drive javiewer porn spider japanese magnet

jaybizzle / crawler-detect
spider,🕷 CrawlerDetect is a PHP class for detecting bots/crawlers/spiders via the user agent
User: jaybizzle
Home Page: https://crawlerdetect.io
bots crawler detect hacktoberfest php spider user-agent

jhao104 / proxy_pool
spider,Python ProxyPool for web spider
User: jhao104
Home Page: https://jhao104.github.io/proxy_pool/
crawler http proxy redis spider

jumper2014 / lianjia-beike-spider
spider,链家网和贝壳网房价爬虫，采集北京上海广州深圳等21个**主要城市的房价数据（小区，二手房，出租房，新房），稳定可靠快速！支持csv,MySQL, MongoDB,Excel, json存储，支持Python2和3，图表展示数据，注释丰富 ，点星支持，仅供学习参考，请勿用于商业用途，后果自负。
User: jumper2014
lianjia spider crawler beike house

kangvcar / infospider
spider,INFO-SPIDER 是一个集众多数据源于一身的爬虫工具箱🧰，旨在安全快捷的帮助用户拿回自己的数据，工具代码开源，流程透明。支持数据源包括GitHub、QQ邮箱、网易邮箱、阿里邮箱、新浪邮箱、Hotmail邮箱、Outlook邮箱、京东、淘宝、支付宝、**移动、**联通、**电信、知乎、哔哩哔哩、网易云音乐、QQ好友、QQ群、生成朋友圈相册、浏览器浏览历史、12306、博客园、CSDN博客、开源**博客、简书。
User: kangvcar
Home Page: https://infospider.vercel.app
automation chrome crawl csdn hotmail outlook python3 selenium spider tkinter wxpython

lorien / awesome-web-scraping
spider,List of libraries, tools and APIs for web scraping and data processing.
User: lorien
web-scraping captcha-bypass captcha-recaptcha crawling crawling-framework crawling-python crawling-tool scraping scraping-framework scraping-python

lorien / grab
spider,Web Scraping Framework
User: lorien
Home Page: https://grab.readthedocs.io
web-scraping http-client framework python pycurl asynchronous network urllib3 spider crawler crawling scraping python-library python3

luyishisi / anti-anti-spider
spider,越来越多的网站具有反爬虫特性，有的用图片隐藏关键数据，有的使用反人类的验证码，建立反反爬虫的代码仓库，通过与不同特性的网站做斗争（无恶意）提高技术。（欢迎提交难以采集的网站）（因工作原因，项目暂停）
User: luyishisi
Home Page: https://www.urlteam.cn
python spider geek

my8100 / scrapydweb
spider,Web app for Scrapyd cluster management, Scrapy log analysis & visualization, Auto packaging, Timer tasks, Monitor & Alert, and Mobile UI. DEMO :point_right:
User: my8100
Home Page: https://github.com/my8100/files
scrapy scrapyd scrapyd-ui scrapyd-api scrapyd-admin scrapyd-manage log-parsing log-analysis scrapyd-monitor scrapyd-keeper

naibowang / easyspider
spider,A visual no-code/code-free web crawler/spider易采集：一个可视化浏览器自动化测试/数据采集/爬虫软件，可以无代码图形化的设计和执行爬虫任务。别名：ServiceWrapper面向Web应用的智能化服务封装系统。
User: naibowang
Home Page: https://www.easyspider.net
batch-processing batch-script code-free crawler data-collection frontend gui html input-parameters layman parameters robotics rpa scraper spider visual visualization visualprogramming web www

nemo2011 / bilibili-api
spider,哔哩哔哩常用API调用。支持视频、番剧、用户、频道、音频等功能。原仓库地址：https://github.com/MoyuScript/bilibili-api
User: nemo2011
Home Page: https://nemo2011.github.io/bilibili-api/
api bilibili bilibili-api python spider

niespodd / browser-fingerprinting
spider,Analysis of Bot Protection systems with available countermeasures 🚿. How to defeat anti-bot system 👻 and get around browser fingerprinting scripts 🕵️‍♂️ when scraping the web?
User: niespodd
Home Page: https://niespodd.github.io/browser-fingerprinting/
bot detection chromium stealth puppeteer scraper webscraping web automation chromium-browser

pibigstar / go-demo
spider,Go语言实例教程从入门到进阶，包括基础库使用、设计模式、面试易错点、工具类、对接第三方等
User: pibigstar
Home Page: https://pibigstar.github.io
blockchain design go go-demo go-design go-utils goutils interview kafaka leetcode oss pprof qq redis spider

qianyantech / image-downloader
spider,Download images from Google, Bing, Baidu. 谷歌、百度、必应图片下载.
Organization: qianyantech
image-downloader google-images bing baidu google pyqt scrapy spider

s0md3v / photon
spider,Incredibly fast crawler designed for OSINT.
User: s0md3v
crawler information-gathering osint python spider

shengqiangzhang / examples-of-web-crawlers
spider,一些非常有趣的python爬虫例子,对新手比较友好,主要爬取淘宝、天猫、微信、微信读书、豆瓣、QQ等网站。(Some interesting examples of python crawlers that are friendly to beginners. )
User: shengqiangzhang
crawler spider taobao tmall example python selenium pyquery stock fund

shiyanhui / dht
spider,BitTorrent DHT Protocol && DHT Spider.
User: shiyanhui
Home Page: http://bthub.io
dht spider bittorrent-dht-protocol go

sjdirect / abot
spider,Cross Platform C# web crawler framework built for speed and flexibility. Please star this project! +1.
User: sjdirect
c-sharp abot abot-nuget crawler web-crawler parsing spider spiders pluggable unit-testing

spiderclub / haipproxy
spider,:sparkling_heart: High available distributed ip proxy pool, powerd by Scrapy and Redis
Organization: spiderclub
Home Page: https://spiderclub.github.io/haipproxy/
high-availability scrapy ipproxy distributed redis crawler scheduler spider

ssssssss-team / spider-flow
spider,新一代爬虫平台，以图形化方式定义爬虫流程，不写代码即可完成爬虫。
Organization: ssssssss-team
Home Page: https://www.spiderflow.org
spider crawler jsoup xpath web-spider webspider webcrawler web-crawler spider-flow

tophubs / toplist
spider,今日热榜，一个获取各大热门网站热门头条的聚合网站，使用Go语言编写，多协程异步快速抓取信息，预览:https://mo.fish
Organization: tophubs
hot golang spider today-s-hot-list hotlist

wechatsync / wechatsync
spider,一键同步文章到多个内容平台，支持今日头条、WordPress、知乎、简书、掘金、CSDN、typecho各大平台，一次发布，多平台同步发布。解放个人生产力
Organization: wechatsync
Home Page: https://www.wechatsync.com?utm_source=github_info
blog chrome chrome-extension markdown multiplatform spider vue wechat-official-account writer

wkunzhi / python3-spider
spider,Python爬虫实战 - 模拟登陆各大网站 包含但不限于：滑块验证、拼多多、美团、百度、bilibili、大众点评、淘宝，如果喜欢请start ❤️
User: wkunzhi
scrapy python crawl crawler geek spider taobao dianping meituan selenium

wnma3mz / wechat_articles_spider
spider,微信公众号文章的爬虫
User: wnma3mz
wechat spider officialaccounts wechat-official-account python36

xianhu / pspider
spider,简单易用的Python爬虫框架，QQ交流群：597510560
User: xianhu
Home Page: https://github.com/xianhu/PSpider
crawler spider python proxies web-spider multi-threading web-crawler python-spider multiprocessing

zorlan / skycaiji
spider,蓝天采集器是一款开源免费的爬虫系统，仅需点选编辑规则即可采集数据，可运行在本地、虚拟主机或云服务器中，几乎能采集所有类型的网页，无缝对接各类CMS建站程序，免登录实时发布数据，全自动无需人工干预！是网页大数据采集软件中完全跨平台的云端爬虫系统
User: zorlan
Home Page: https://www.skycaiji.com
crawler crawling php spider webcrawler