jsdom / html-encoding-sniffer Goto Github PK

Sniff the encoding from a HTML byte stream

License: MIT License

JavaScript 100.00%

html-encoding-sniffer's Introduction

Determine the Encoding of a HTML Byte Stream

This package implements the HTML Standard's encoding sniffing algorithm in all its glory. The most interesting part of this is how it pre-scans the first 1024 bytes in order to search for certain <meta charset>-related patterns.

const htmlEncodingSniffer = require("html-encoding-sniffer");
const fs = require("fs");

const htmlBytes = fs.readFileSync("./html-page.html");
const sniffedEncoding = htmlEncodingSniffer(htmlBytes);

The passed bytes are given as a Uint8Array; the Node.js Buffer subclass of Uint8Array will also work, as shown above.

The returned value will be a canonical encoding name (not a label). You might then combine this with the whatwg-encoding package to decode the result:

const whatwgEncoding = require("whatwg-encoding");
const htmlString = whatwgEncoding.decode(htmlBytes, sniffedEncoding);

Options

You can pass two potential options to htmlEncodingSniffer:

const sniffedEncoding = htmlEncodingSniffer(htmlBytes, {
  transportLayerEncodingLabel,
  defaultEncoding
});

These represent two possible inputs into the encoding sniffing algorithm:

transportLayerEncodingLabel is an encoding label that is obtained from the "transport layer" (probably a HTTP Content-Type header), which overrides everything but a BOM.
defaultEncoding is the ultimate fallback encoding used if no valid encoding is supplied by the transport layer, and no encoding is sniffed from the bytes. It defaults to "windows-1252", as recommended by the algorithm's table of suggested defaults for "All other locales" (including the en locale).

Credits

This package was originally based on the excellent work of @nicolashenry, in jsdom. It has since been pulled out into this separate package.

html-encoding-sniffer's People

Contributors

Stargazers

Watchers

Forkers

abromeit amishne zbinlin zirro openandclose fengkx aricgit rakhithjk mpadev0103 popular-dependencies seanpm2001

html-encoding-sniffer's Issues

Why does it get this page wrong?

wget https://ehoba.hatenablog.com/entry/2020/10/25/190724

sourcepath = 190724

const htmlEncodingSniffer = require("html-encoding-sniffer")
const fs = require("fs");

const htmlBytes = fs.readFileSync(sourcepath);
const sniffedEncoding = htmlEncodingSniffer(htmlBytes);
console.log(sniffedEncoding) // "windows-1252"

The html source is kinda messy, but there is a <meta charset="utf-8"/> in there. Firefox renders the correct utf-8 charset.

Revamped package

Hi @domenic et al,

I was looking into using this package for cheerio, but had several issues that I wanted to fix first. As extending the current html-encoding-sniffer package turned out to be cumbersome, I opted to write a new module instead:

https://github.com/fb55/encoding-sniffer

This new package implements the current version of the encoding sniffing algo as a state machine. That allows streams to be supported without much effort. Features this supports, which aren't present in html-encoding-sniffer:

XML encoding types (UTF-16 prefixes and <?xml encoding="...">)
Configurable sniff depth (see #10)
x-user-defined in <meta> tags (turns out html-encoding-sniffer's support is broken)
Streams / sniffing of incomplete documents

I would love to join forces and have a single package that both jsdom and cheerio can use going forward. Let me know if this is something you'd be interested in!

Fail to parse short bogus comments

Even in prescan, abrupt-closing-of-empty-comment should be treated as normal comments.

for <!-->, the spec states rather explicitly.

The two 0x2D bytes can be the same as those in the '<!--' sequence.'

But I think it is also applied to <!--->.

A failure test example, just in case.
openandclose@0370865

prescan:
https://html.spec.whatwg.org/multipage/parsing.html#prescan-a-byte-stream-to-determine-its-encoding
abrupt-closing-of-empty-comment:
https://html.spec.whatwg.org/multipage/parsing.html#parse-error-abrupt-closing-of-empty-comment

Add an option to change prescanning byte length

Just thought I'd make this suggestion because it looks pretty easy to implement and sites that use odd encodings don't always follow best practices with meta tags

Fix bug: content attribute charset value, trailing whitespace

The following example should return 'ISO-8859-2'
(note the last whitespace before '"').

<meta http-equiv="Content-Type" content="text/html; charset=iso8859-2 ">

While html-encoding-sniffer returns 'defaultEncoding',
calling whatwgEncoding.labelToName with l; charset=.

line 283:
let end = string.substring(position + 1).search(/\x09|\x0A|\x0C|\x0D|\x20|;/);

end gets the index of the shortened string.

Fix bug: content attribute, second charset index

The following example should return 'ISO-8859-2'.

<meta http-equiv="Content-Type" content="charsetcharset=iso-8859-2">

While html-encoding-sniffer freezes.

line 242:
let subPosition = string.substring(position).search(/charset/i);

subPosition gets the index of the shortened string
(from second iteration or third, always 1).

ease adoption by enterprises

WTFPL shows “lack of warranty disclaimer, very vague rights grant”, according to Google’s recommendation for employee contributors, https://opensource.google.com/docs/thirdparty/licenses/#wtfpl-not-allowed

I wonder if MIT or another top-5 OSI approved license could help in adopting this npm module by enterprises?

Fix bug: second and invalid http-equiv

The following example should return 'null' or default encoding.
The spec uses 'attribute list', to keep track of parsed attribute names.

<meta http-equiv="refresh" http-equiv="Content-Type" content="text/html; charset=iso8859-2">

While html-encoding-sniffer returns 'ISO-8859-2'.