Comments (3)
kostalski
commented on August 24, 2024
3
Hi @samuelchen @Gallaecio ,
Source of encoding detection problem seems to be in invalid input HTML it self not in w3lib. There is invalid HTML meta tag. There is <meta httpequiv="ContentType" ..., but valid (with w3c) should be <meta http-equiv="Content-Type" ...(missing dash character). Because of that w3lib is not detecting defined encoding.
beautifulsoup4 is detecting 'gbk' encoding, because it is using naive regex for fallback encoding detection (lib: beautifulsoup4 file: bs4/dammit.py, line: html_meta = '<\\s*meta[^>]+charset\\s*=\\s*["\']?([^>]*?)[ /;\'">]').
For @samuelchen problem w3lib can be updated to be more forgiving/lenient. Updating (lib: w3lib, file: w3lib/encoding.py)
From: _HTTPEQUIV_RE = _TEMPLATE % ('http-equiv', 'Content-Type')
To: _HTTPEQUIV_RE = _TEMPLATE % (r'http-?equiv', r'Content-?Type')
After this fix w3lib would detected encoding as gb18030. This should have no side effects, but I don't know if it is right way ;)
What you think @Gallaecio ?
More details below.
Details
I was able to reproduce issue with provided settings:
- Python 3.7.9
- libs:
-- beautifulsoup4==4.9.3
-- html5lib==1.1
-- lxml==4.6.1
-- w3lib==1.22.0
Test python script:
from w3lib.encoding import html_body_declared_encoding
from bs4 import BeautifulSoup
b = b'<HTML>\r\n <HEAD>\r\n <TITLE>\xcd\xf8\xd5\xbe\xb5\xd8\xcd\xbc</TITLE>\r\n <meta httpequiv="ContentType" content="text/html; charset=gbk" />\r\n <META NAME="Keywords" CONTENT="\xe5\xd0\xa1\xcb\xb5,\xcd\xf8\xd5\xbe\xb5\xd8\xcd\xbc">'
enc = html_body_declared_encoding(b)
print("html_body_declared_encoding: %s" % enc)
for parser in ['html5lib', 'html.parser', 'lxml']:
soup = BeautifulSoup(b, parser)
print("soup.original_encoding[parser:{}]: {}".format(parser, soup.original_encoding))
Script output:
html_body_declared_encoding: None
soup.original_encoding[parser:html5lib]: windows-1252
soup.original_encoding[parser:html.parser]: windows-1252
soup.original_encoding[parser:lxml]: gbk
Detection by Beatifulsoup only for 'lxml' parser, by fallback encoding detection.
lib: beautifulsoup4
file: bs4/dammit.py,
line: html_meta = '<\\s*meta[^>]+charset\\s*=\\s*["\']?([^>]*?)[ /;\'">]'
from w3lib.
kostalski
commented on August 24, 2024
1
Ok @samuelchen, no problem 👍
from w3lib.
samuelchen
commented on August 24, 2024
@kostalski Thank you for the feedback. I am not able to recall why that html was httpequiv="ContentType". Not sure if it is possible to be coverted by other parts of scrapy or it's original. I am sorry about this, too long ago to remember that.
btw. GB18030 is compatible with GBK.
from w3lib.
Related Issues (20)
- url.add_or_replace_parameter(s) removes param values for param with multiple values HOT 2
- remove_tags not working on html comments HOT 4
- Redirection may not work depending on order of 'content' and 'http-equiv' in meta tag HOT 5
- Fix CI issue on PyPy 3 and the Rust compiler HOT 1
- test_add_or_replace_parameter fails on Python 3.6.13, 3.7.10, 3.8.8, 3.9.2 due to CVE-2021-23336 fix HOT 1
- w3lib.url.safe_url_string incorrectly encode IDNA domain with port HOT 1
- should the canonize_url function convert an apostrophe to %27 HOT 2
- [request] Update pypi release HOT 1
- basic_auth_header uses the wrong flavor of base64
- safe_url_string URL-encodes already-encoded username and password, breaking idempodency HOT 1
- Issue in safe_url_encoding HOT 8
- BOM should take precedence over Content-Type header when detecting the encoding
- safe_url_string handling IPv6 URLs HOT 5
- Function `convert_entity` does not catch `OverflowError`
- Reimplement safe_url_string based on the URL living standard HOT 4
- Location of libw3.a HOT 1
- test_safe_url_string_url regressed on 3.11.4 HOT 1
- canonicalize_url incorrectly handles port when using hostname that requires IDNA encoding HOT 1
- Space at end of query string is trimmed HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from w3lib.