Comments (6)
Hi Nathan,
At the moment you might try to set encoding manually (just copy the __main__
code from here https://github.com/buriy/python-readability/blob/master/readability/readability.py#L588 to your file if you use it from the command-line) or to write some code to extract meta-charset information from the page (please note this information could provide wrong encoding sometimes).
I'll happily merge it into this branch if you or someone else will contribute this functionality.
There's also related unsolved #42 related to finding the best solution for this problem at large.
from python-readability.
I made a pull request with some specific handling of Chinese encodings (gb2312 is often NOT gb2312).
For the content issue, poor Chinese web design in general makes it difficult to correctly determine content. In the example you posted, they use tables to organize content. It's difficult to use this readability algorithm on typical Chinese websites for this sort of reason. This difficulty probably won't go away without large changes. At my company we scrape large amounts of Chinese text and have basically decided to just live with the fact that many sites are unscrapable, and putting in special handling for large/important sites.
from python-readability.
Thanks a lot, @mperdomo1
I'll try to test it on some Chinese news sources tomorrow and will merge if everything will go well.
Have you got a chance to compare python-readability to newspaper and python-goose projects yet?
from python-readability.
I checked out newspaper when it was new and it seemed like it used python-goose to handle readability, and at the time when I looked at it, python-readability was a lot more robust and better at picking out content. This definitely could have changed though. I am actually thinking about writing a readability project dedicated to CJK pages because the structure can often be so broken.
from python-readability.
@mperdomo1 Thanks, I've applied your patch, but for the document mentioned by Nathan it's still blank.
@nathanathan
I'm working on resolving your issue and also rectify logging in new version.
Found a serious html sanitise error related to the images block included into (parent) content block -- the content block just gets removed, only images block should be removed instead.
P.S. Also I'll have a chance to test readability on millions of URLs from thousand of news sources in several next months, so I'll add a lot of new tests and we'll get a completely refreshed version of the module by then.
from python-readability.
The article in question works fine now. So closing this issue.
Thanks for patience.
from python-readability.
Related Issues (20)
- Orphan links in doc.summary()
- Inlining images?
- re._pattern_type has been removed in favor of re.Pattern in Python 3.7
- No chance for GitHub commit page? HOT 1
- RuntimeWarning and Correct invocation on the shell command line (not python script) HOT 1
- Missing <p>-text
- REGEXES["divToPElementsRe"] logical error HOT 3
- Error when using positive_keywords (or negative_keywords) argument with python >= 3.7 HOT 1
- .text may guess the encoding incorrectly HOT 4
- Does not handle github pages
- <p> wrongly inserted before <i> or <b>
- Problems with thecyberwire.com
- Circular import error for pip install readability-lxml HOT 4
- isProbablyReaderable HOT 3
- Issue with utf8 and HTML entities HOT 2
- Last two commands in the "usage" section are incorrect
- `test_many_repeated_spaces` fails on darwin python 3.8, 3.9 & 3.10
- Consider switching from lxml's clean_html for enhanced security (and possibly performance) HOT 5
- Summary is fooled by a modal popup
- Readability of MSN articles
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from python-readability.