Comments (2)
爬虫不会抛弃爬取失败的URL,默认情况下, 每一层重试3次,整个流程中重试Config.MAX_RETRY次。默认的Generator会过滤超过Config.MAX_RETRY次爬取失败的链接。
只要在爬取时,将Config.MAX_RETRY设置为一个较大的数就行了。
from webcollector.
我的做法和题主的做法是一样的。不过我拿到过100多个nextLinks。但是确实也感觉有点不放心的感觉,很担心会出异常。出异常了再过来问大神哈。
from webcollector.
Related Issues (20)
- 爬取一段时间后总是会抛出RocksDBException异常,不清楚什么原因。 HOT 2
- 发布包里包含log4j配置文件,会覆盖别人的log4j配置文件 HOT 3
- 前端渲染的页面怎么样使用webcollector进行爬取数据
- 继承BreadthCrawler,获取网页中文部分输出乱码 HOT 2
- 抛异常的日志级别能不能改warn或error
- 请问如何在爬虫内部解决数据库连接过多的问题
- 2.73-alpha版如何忽略https证书过期问题? HOT 2
- 亲问下分布式的版本在哪里
- 请问执行教程中的爬取CSDN博客原码出错 HOT 1
- ContentExtractor中的computeInfo函数会出现StackOverflowError HOT 3
- 运行爬取CSDN示例代码时,出现RocksDBException,Failed to create a directory: C:\code\weibocrawler\crawl\crawldb: ϵͳÕҲ»µ½ָ¶ HOT 3
- 抽取时间的正则在时那点应该改成【0-9】?
- out of memory 问题。
- 访问连接307怎么处理啊
- 请问论文中的准确度是如何计算的?
- 大哥些 官方群都加不了了啊。全都提示满了 HOT 1
- 自动识别新闻时间部分存在BUG HOT 1
- ContentExtractor.getContentByUrl返回的内容没有空行等格式排版 HOT 1
- Inefficient code detected in RegexRule.java HOT 1
- 访问的页面报502异常,但是还需要访问,visit异常设置了ExceptionUtils.fail(e)还是不行,怎么解决 HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from webcollector.