The fastest smallest Javascript polyfill for encodeInto of TextEncoder, encode of TextEncoder, and decode of TextDecoder for UTF-8 only.
Home Page: https://anonyco.github.io/FastestSmallestTextEncoderDecoder/gh-pages/
License: Creative Commons Zero v1.0 Universal
To support Worklets we have to consider globalThis.
Basically:
})(typeof global == "" + void 0 ? typeof self == "" + void 0 ? typeof this == "" + void 0 ? globalThis : this : self : global);<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8">
</head>
<body>
<script src="https://dl.dropboxusercontent.com/s/r55397ld512etib/EncoderDecoderTogether.min.js?dl=0" nomodule="" type="text/javascript"></script>
<script type="text/javascript">
const bytes = [50,65,113,117,121,81,111,98,118,68,76,43,77,110,73,90,49,100,43,77,65,71,119,87,68,82,115,57,74,54,117,97,79,78,120,74,119,54,88,113,120,86,99];
var allBytes = new Uint8Array(16777216);
// write some A's to the beginning
for (var i = 100 - 1; i >= 0; i--) {
allBytes[i] = 65;
}
const offset = 242839;
for (var i = bytes.length - 1; i >= 0; i--) {
allBytes[i + offset] = bytes[i];
}
const slice = allBytes.subarray(offset, offset + bytes.length);
console.log("slice", slice.length, slice[0], slice);
const str = new TextDecoder().decode(slice);
console.log("str", str.length, str);
</script>
</body>
</html>Output on IE11:
slice 43 50 [object Uint8Array]
str 16777216 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
Output in Firefox 80 (expected output):
slice 43 50 Uint8Array(43) [ 50, 65, 113, 117, 121, 81, 111, 98, 118, 68, … ]
str 43 2AquyQobvDL+MnIZ1d+MAGwWDRs9J6uaONxJw6XqxVc
As you can see, on IE11, the slice returned from subarray has the correct length and correct first byte, but somehow decode decodes the entire allBytes array (until it finds a 0 byte?).
I think this polyfill behavior is different from the native version for the function decode of TextDecoder.
When using new TextDecoder.decode(); with no argument :
This polyfill will throw a blocking error, whereas using the native one returns an empty string.
I believe this property is required as part of the specification -- https://encoding.spec.whatwg.org/#dom-textencoder-encoding but I can not see it being set in this polyfill.
new TextEncoder().encoding === 'utf-8'We are using this library as a polyfill for IE11 which appears to not have TextEncoder defined.
We are using Angular 8.
I previously had version 1.0.8 installed, and in my polyfills.ts I had import 'fastestsmallesttextencoderdecoder'; and all was well in Chrome and Firefox, but IE11 had a problem with that, with TextEncoder being undefined.
After updating to 1.0.14 and making no other changes, I was getting errors from the library that f.decode is undefined. I think it had to do with the version of the library that was being loaded in the browser, it appeared to either be incompatible with Chrome/Firefox or the build process was using the wrong file (perhaps it was using the node file in the browser?).
After changing my import to specifically target the browser version, i.e. import 'fastestsmallesttextencoderdecoder/EncoderDecoderTogether.min'; in my polyfills.ts, all is well AND IE11 is now working just fine.
I used nvm to verify it works on 0.10 and 0.12, but then breaks on node 4 or higher. https://nodejs.org/en/download/releases/
asa:~/repos$ cat test.js
var tmp = require('fastestsmallesttextencoderdecoder')
console.log(
tmp.decode(tmp.encode("Hello"))
)
asa:~/repos$ nvm install 0.12; nvm use 0.12; node --version; node test.js
Downloading and installing node v0.12.18...
Downloading https://nodejs.org/dist/v0.12.18/node-v0.12.18-linux-x64.tar.xz...
########################################################################################################################################################################################################### 100.0%
Computing checksum with sha256sum
Checksums matched!
Now using node v0.12.18 (npm v2.15.11)
Now using node v0.12.18 (npm v2.15.11)
v0.12.18
Hello
asa:~/repos$ nvm install 4; nvm use 4; node --version; node test.js
v4.9.1 is already installed.
Now using node v4.9.1 (npm v2.15.11)
Now using node v4.9.1 (npm v2.15.11)
v4.9.1
/home/asa/repos/node_modules/fastestsmallesttextencoderdecoder/NodeJS/EncoderAndDecoderNodeJS.min.js:2
A);for(var a=b.length|0,c=new (h?k:e)(a),d=0;d<a;d=d+1|0)c[d]=b.charCodeAt(d)|0;return c},q=function(){},A=function(b){var a=b.charCodeAt(0)|0;if(55296<=a&&56319>=a){var c=b.charCodeAt(1)|0;if(c===c&&56320<=c&&57343>=c){if(a=(a-55296<<10)+c-56320+65536|0,65535<a)return g(240|a>>>18,128|a>>>12&63,128|a>>>6&63,128|a&63)}else return g(239,191,189)}return 127>=a?b:2047>=a?g(192|a>>>6,128|a&63):g(224|a>>>12,128|a>>>6&63,128|a&63)},r=function(b){b=b&&b.buffer||b;var a=m.call(b);if(a!==z&&a!==y)throw Error("Failed to execute 'decode' on 'TextDecoder': The provided value is not of type '(ArrayBuffer or ArrayBufferView)'");
Error: Failed to execute 'decode' on 'TextDecoder': The provided value is not of type '(ArrayBuffer or ArrayBufferView)'
asa:~/repos$ nvm install 13; nvm use 13; node --version; node test.js
v13.5.0 is already installed.
Now using node v13.5.0 (npm v6.13.4)
Now using node v13.5.0 (npm v6.13.4)
v13.5.0
/home/asa/repos/node_modules/fastestsmallesttextencoderdecoder/NodeJS/EncoderAndDecoderNodeJS.min.js:2
A);for(var a=b.length|0,c=new (h?k:e)(a),d=0;d<a;d=d+1|0)c[d]=b.charCodeAt(d)|0;return c},q=function(){},A=function(b){var a=b.charCodeAt(0)|0;if(55296<=a&&56319>=a){var c=b.charCodeAt(1)|0;if(c===c&&56320<=c&&57343>=c){if(a=(a-55296<<10)+c-56320+65536|0,65535<a)return g(240|a>>>18,128|a>>>12&63,128|a>>>6&63,128|a&63)}else return g(239,191,189)}return 127>=a?b:2047>=a?g(192|a>>>6,128|a&63):g(224|a>>>12,128|a>>>6&63,128|a&63)},r=function(b){b=b&&b.buffer||b;var a=m.call(b);if(a!==z&&a!==y)throw Error("Failed to execute 'decode' on 'TextDecoder': The provided value is not of type '(ArrayBuffer or ArrayBufferView)'");
Error: Failed to execute 'decode' on 'TextDecoder': The provided value is not of type '(ArrayBuffer or ArrayBufferView)'
Hey Jack!
Thank you again for this amazing library!
I'n using parts of the code in https://github.com/Bnaya/objectbuffer
V8 and other js engines are trying to deduplicate strings references, and also hold strings as rope data structure.
In my use-case, i'm doing many decoding of the same string over and over again, and i'm not sure how to actual memory in the js engine side gonna end up.
I was wondering if you made any analysis regarding that topic and if you cloud share some findings
Thanks!
Bnaya
IE 11 doesn't support Uint8Array.prototype.slice.
Here is the warning:
WARNING in /app/node_modules/fastestsmallesttextencoderdecoder/EncoderDecoderTogether.min.js
Module Warning (from /app/node_modules/source-map-loader/index.js):
(Emitted value instead of an instance of Error) Cannot find SourceMap 'https://cdn.jsdelivr.net/gh/AnonyCo/FastestSmallestTextEncoderDecoder/EncoderDecoderTogether.min.js.map': Error: Can't resolve './https://cdn.jsdelivr.net/gh/AnonyCo/FastestSmallestTextEncoderDecoder/EncoderDecoderTogether.min.js.map' in '/app/node_modules/fastestsmallesttextencoderdecoder'
....
@ ./src/main.ts
Note that the url starts with './https://cdn...'
In version 1.0.4 calling the text encoder like
new TextEncoder('utf-8').encode(value).length
causes the following error
TypeError: fastestsmallesttextencoderdecoder__WEBPACK_IMPORTED_MODULE_0__.TextEncoder is not a constructor
This error does not occur in v1.0.3
We noticed encoding issues using this library in production, and after some investigation it turns out there's a correctness issue in the library. I took the official WPT testcases and threw a loose harness around them to make them runnable:
// META: global=window,worker
// META: script=/common/sab.js
delete TextEncoder;
require('fastestsmallesttextencoderdecoder-encodeinto/EncoderDecoderTogether.min');
self = globalThis;
function createBuffer(t, s) {
return new ArrayBuffer(s);
}
function assert_equals(a, b) {
if (a !== b) {
throw new Error(`Assertion failed: ${a} is not equal to ${b}`);
}
}
function assert_throws_js(e, c) {
try {
c();
} catch (err) {
if (!(err instanceof e)) {
throw err;
}
}
}
function test(t, d) {
try {
t();
} catch (e) {
console.log('Test failed with error', e);
}
}
[
{
input: 'Hi',
read: 0,
destinationLength: 0,
written: [],
},
{
input: 'A',
read: 1,
destinationLength: 10,
written: [0x41],
},
{
input: '\u{1D306}', // "\uD834\uDF06"
read: 2,
destinationLength: 4,
written: [0xf0, 0x9d, 0x8c, 0x86],
},
{
input: '\u{1D306}A',
read: 0,
destinationLength: 3,
written: [],
},
{
input: '\uD834A\uDF06A¥Hi',
read: 5,
destinationLength: 10,
written: [0xef, 0xbf, 0xbd, 0x41, 0xef, 0xbf, 0xbd, 0x41, 0xc2, 0xa5],
},
{
input: 'A\uDF06',
read: 2,
destinationLength: 4,
written: [0x41, 0xef, 0xbf, 0xbd],
},
{
input: '¥¥',
read: 2,
destinationLength: 4,
written: [0xc2, 0xa5, 0xc2, 0xa5],
},
].forEach((testData) => {
[
{
bufferIncrease: 0,
destinationOffset: 0,
filler: 0,
},
{
bufferIncrease: 10,
destinationOffset: 4,
filler: 0,
},
{
bufferIncrease: 0,
destinationOffset: 0,
filler: 0x80,
},
{
bufferIncrease: 10,
destinationOffset: 4,
filler: 0x80,
},
{
bufferIncrease: 0,
destinationOffset: 0,
filler: 'random',
},
{
bufferIncrease: 10,
destinationOffset: 4,
filler: 'random',
},
].forEach((destinationData) => {
['ArrayBuffer', 'SharedArrayBuffer'].forEach((arrayBufferOrSharedArrayBuffer) => {
test(() => {
// Setup
const bufferLength = testData.destinationLength + destinationData.bufferIncrease;
const destinationOffset = destinationData.destinationOffset;
const destinationLength = testData.destinationLength;
const destinationFiller = destinationData.filler;
const encoder = new TextEncoder();
const buffer = createBuffer(arrayBufferOrSharedArrayBuffer, bufferLength);
const view = new Uint8Array(buffer, destinationOffset, destinationLength);
const fullView = new Uint8Array(buffer);
const control = new Array(bufferLength);
let byte = destinationFiller;
for (let i = 0; i < bufferLength; i++) {
if (destinationFiller === 'random') {
byte = Math.floor(Math.random() * 256);
}
control[i] = byte;
fullView[i] = byte;
}
// It's happening
const result = encoder.encodeInto(testData.input, view);
// Basics
assert_equals(view.byteLength, destinationLength);
assert_equals(view.length, destinationLength);
// Remainder
assert_equals(result.read, testData.read);
assert_equals(result.written, testData.written.length);
for (let i = 0; i < bufferLength; i++) {
if (i < destinationOffset || i >= destinationOffset + testData.written.length) {
assert_equals(fullView[i], control[i]);
} else {
assert_equals(fullView[i], testData.written[i - destinationOffset]);
}
}
}, 'encodeInto() into ' + arrayBufferOrSharedArrayBuffer + ' with ' + testData.input + ' and destination length ' + testData.destinationLength + ', offset ' + destinationData.destinationOffset + ', filler ' + destinationData.filler);
});
});
});
[
'DataView',
'Int8Array',
'Int16Array',
'Int32Array',
'Uint16Array',
'Uint32Array',
'Uint8ClampedArray',
'BigInt64Array',
'BigUint64Array',
'Float32Array',
'Float64Array',
].forEach((type) => {
['ArrayBuffer', 'SharedArrayBuffer'].forEach((arrayBufferOrSharedArrayBuffer) => {
test(() => {
const viewInstance = new self[type](createBuffer(arrayBufferOrSharedArrayBuffer, 0));
assert_throws_js(TypeError, () => new TextEncoder().encodeInto('', viewInstance));
}, 'Invalid encodeInto() destination: ' + type + ', backed by: ' + arrayBufferOrSharedArrayBuffer);
});
});
['ArrayBuffer', 'SharedArrayBuffer'].forEach((arrayBufferOrSharedArrayBuffer) => {
test(() => {
assert_throws_js(TypeError, () =>
new TextEncoder().encodeInto('', createBuffer(arrayBufferOrSharedArrayBuffer, 10)),
);
}, 'Invalid encodeInto() destination: ' + arrayBufferOrSharedArrayBuffer);
});
test(() => {
const buffer = new ArrayBuffer(10),
view = new Uint8Array(buffer);
let { read, written } = new TextEncoder().encodeInto('', view);
assert_equals(read, 0);
assert_equals(written, 0);
new MessageChannel().port1.postMessage(buffer, [buffer]);
({ read, written } = new TextEncoder().encodeInto('', view));
assert_equals(read, 0);
assert_equals(written, 0);
({ read, written } = new TextEncoder().encodeInto('test', view));
assert_equals(read, 0);
assert_equals(written, 0);
}, 'encodeInto() and a detached output buffer');
This then fails with a bunch of failed a assertions: Test failed with error Error: Assertion failed: 1 is not equal to 2. Commenting out the delete TextEncoder line then allows us to verify that the tests pass when executed with the native TextEncoder.
This would enable debugging the package when there's a problem.
This would make tree-shaking in Webpack not include the same amalgamated file twice when used in ProvidePlugin.
When an out of order surrogate is encountered it should be replaced with the replacement character (utf-8 bytes 0xef 0xbf 0xbd).
Example:
console.log((new TextEncoder()).encode("\uDC00"));
result:
Uint8Array(3) [ 237, 176, 128 ]
expected:
Uint8Array(3) [ 239, 191, 189 ]
At https://github.com/anonyco/FastestSmallestTextEncoderDecoder/blob/master/individual/FastestTextDecoderPolyfill.src.js#L52 the input Uint8Array sometimes carry a non-zero byteOffset, which is needed to get the correct underlying ArrayBuffer.
> var aaa = new Uint8Array(new TextEncoder().encode("wrong hello").buffer, 6);
> new TextDecoder().decode(aaa);
"hello" // Chrome native
> new TextDecoderPolyfill().decode(aaa);
"wrong hello" // polyfill
A simple fix could be
var buffer = (inputArrayOrBuffer && inputArrayOrBuffer.buffer && inputArrayOrBuffer.buffer.slice(inputArrayOrBuffer.byteOffset, inputArrayOrBuffer.byteOffset + inputArrayOrBuffer.byteLength)) || inputArrayOrBuffer;
instead
I'm executing this polyfill in v8, and getting:
TypeError: Cannot read property 'allocUnsafe' of undefined
It's coming from this line:
var NativeBuffer_allocUnsafe = NativeBuffer["allocUnsafe"];
Commenting out that line appears to fix the problem. Perhaps setting up NativeBuffer_allocUnsafe should be guarded by some other conditions? (My javascript is not strong.)
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.