Comments (5)
你那是网页404了吧
from webcollector.
不是, 你可以看一下这个网页,是正常的http://36kr.com/p/5043670.html
from webcollector.
1.有些正常的网页爬取会直接出现FileNotFoundException,如http://36kr.com/p/5043670.html
2.上述网页正文的标题并不在<title>,
等标签中,而是在中,如果能访问的话, 应该也不能获取到正确的标题吧
3.获取到网页正文后,过滤出
时会出现有些
没有获取到,如http://www.huxiu.com/article/121959/1.html, 过滤出正文的
,里面没有得到
from webcollector.
FileNotFoundException麻烦贴一下日志,最好能看到异常所在行,谢谢
from webcollector.
1)标题抽取现在是简单启发规则,还没有用比较稳定的算法。
2)目前封装的正文抽取算法目前是尽量保证正文文本被包含,应用场景主要是舆情和搜索。你可以根据自己的业务调整算法,例如你希望做转载,就可以调整参数,或者加入启发式规则,让这种图片尽量被包含。
from webcollector.
Related Issues (20)
- 爬取一段时间后总是会抛出RocksDBException异常,不清楚什么原因。 HOT 2
- 发布包里包含log4j配置文件,会覆盖别人的log4j配置文件 HOT 3
- 前端渲染的页面怎么样使用webcollector进行爬取数据
- 继承BreadthCrawler,获取网页中文部分输出乱码 HOT 2
- 抛异常的日志级别能不能改warn或error
- 请问如何在爬虫内部解决数据库连接过多的问题
- 2.73-alpha版如何忽略https证书过期问题? HOT 2
- 亲问下分布式的版本在哪里
- 请问执行教程中的爬取CSDN博客原码出错 HOT 1
- ContentExtractor中的computeInfo函数会出现StackOverflowError HOT 3
- 运行爬取CSDN示例代码时,出现RocksDBException,Failed to create a directory: C:\code\weibocrawler\crawl\crawldb: ϵͳÕҲ»µ½ָ¶ HOT 3
- 抽取时间的正则在时那点应该改成【0-9】?
- out of memory 问题。
- 访问连接307怎么处理啊
- 请问论文中的准确度是如何计算的?
- 大哥些 官方群都加不了了啊。全都提示满了 HOT 1
- 自动识别新闻时间部分存在BUG HOT 1
- ContentExtractor.getContentByUrl返回的内容没有空行等格式排版 HOT 1
- Inefficient code detected in RegexRule.java HOT 1
- 访问的页面报502异常,但是还需要访问,visit异常设置了ExceptionUtils.fail(e)还是不行,怎么解决 HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from webcollector.