101arrowz / fflate Goto Github PK
View Code? Open in Web Editor NEWHigh performance (de)compression in an 8kB package
Home Page: https://101arrowz.github.io/fflate
License: MIT License
High performance (de)compression in an 8kB package
Home Page: https://101arrowz.github.io/fflate
License: MIT License
Hi everyone,
I am trying to use the library as an ESM module like that:
import * as fflate from 'fflate';
But I am getting Error: Could not resolve import "worker_threads" in "../../../node_modules/fflate/esm/node-worker.js".
In esm/index.js
I see the following line import wk from './node-worker';
and if I change it to import wk from './worker';
it works fine.
As I understand it tries to run node version worker in web for some reason. Does it miss some sort of condition to import proper worker?
I really need this so I am ready to work out a PR for that.
In the demo:
// With fflate, we can choose which files we want to compress
zipObj[file.name] = [buf, {
level: ALREADY_COMPRESSED.indexOf(ext) == -1 ? 6 : 0
}];
Changing 6
to 1
or to any other number except 0
doesn't affect the execution time or file size.
You have to change it here:
if (!--left) {
fflate.zip(zipObj, {
// If you want to control options for every file, you can do so here
// They are merged with the per-file options (if they exist)
// mem: 9
level: 1
}, function(err, out) {
I see in the code, that you do merge them like it says in the comment, but in the following line instead of using the merged result you pass the default options:
Line 2149 in ffbb65c
compress by golang
main.go
package main
import (
"bytes"
"compress/flate"
"os"
)
func main() {
data, err := encode([]byte("hello,world!"))
if err != nil {
panic(err)
}
os.WriteFile("1.data", data, 0666)
}
func encode(data []byte) ([]byte, error) {
var buf = bytes.NewBuffer(nil)
var w, err = flate.NewWriter(buf, flate.DefaultCompression)
if err != nil {
return nil, err
}
defer w.Close()
_, err = w.Write(data)
if err != nil {
return nil, err
}
err = w.Flush()
return buf.Bytes(), err
}
decompress
const fflate = require('fflate');
const fs = require('fs')
const data = fs.readFileSync('1.data')
console.info(fflate.inflateSync(data))
got error unexpected EOF
Where's a distributable for the browser that doesn't require some 3rd party crap like skypack or jsdelivr? I want to host it myself. I don't want to compile packages with node/npm tools. Do something like this: https://github.com/Stuk/jszip/tree/master/dist
or https://github.com/nodeca/pako/tree/master/dist
But you say your software works better than the others and supports what I really want (zlib not zip), so I'd like to give it a try, but you don't make it easy.
They're ugly. Opening this issue as a reminder to do this at some point.
Background
I'm maintaining Threebox, and I updated the repo to Three.js r127 which now uses fflate
in many different loaders.
The problem
The problem is that I'm finding issues to bundle fflate.min.js
through different bundling tools, including browserify, webpack and vue-cli-service.
How to reproduce
Using the file you provided to Three.js fflate.min.js
, referenced from [FBXLoader
] that is also bundled in my plugin, and then bundlef with broserify or webpack (I have faced issues also with vue, but there could be ther more dependencies...) I get the following error every time I execute the bundle commands:
> browserify -p tinyify exports.js > dist/threebox.min.js
Error: Can't walk dependency graph: ENOENT: no such file or directory, lstat 'C:\.....\threebox-new\worker_threads' required by C:\.....\threebox-new\src\objects\fflate.min.js[threebox-new] run build successfully completed
WARNING in ./node_modules/threebox-plugin/src/objects/fflate.min.js 6:538-570
Module not found: Error: Can't resolve 'worker_threads' in 'C:\....\node_modules\threebox-plugin\src\objects'
resolve 'worker_threads' in 'C:\....\node_modules\threebox-plugin\src\objects'
Parsed request is a module
using description file: C:\....\node_modules\threebox-plugin\package.json (relative path: ./src/objects)
Field 'browser' doesn't contain a valid alias configuration
resolve as module
looking for modules in C:/..../node_modules
single file module
using description file: C:\....\package.json (relative path: ./node_modules/worker_threads)
no extension
Field 'browser' doesn't contain a valid alias configuration
C:\....\node_modules\worker_threads doesn't exist
.js
Field 'browser' doesn't contain a valid alias configuration
C:\....\node_modules\worker_threads.js doesn't exist
.json
Field 'browser' doesn't contain a valid alias configuration
C:\....\node_modules\worker_threads.json doesn't exist
.wasm
Field 'browser' doesn't contain a valid alias configuration
C:\....\node_modules\worker_threads.wasm doesn't exist
C:\....\node_modules\worker_threads doesn't exist
ModuleNotFoundError: Module not found: Error: Can't resolve 'worker_threads' in 'C:\....\node_modules\threebox-plugin\src\objects'
at C:\....\node_modules\webpack\lib\Compilation.js:1668:28
at C:\....\node_modules\webpack\lib\NormalModuleFactory.js:712:13
at eval (eval at create (C:\....\node_modules\tapable\lib\HookCodeFactory.js:33:10), <anonymous>:10:1)
at C:\....\node_modules\webpack\lib\NormalModuleFactory.js:273:22
at eval (eval at create (C:\....\node_modules\tapable\lib\HookCodeFactory.js:33:10), <anonymous>:9:1)
at C:\....\node_modules\webpack\lib\NormalModuleFactory.js:402:22
at C:\....\node_modules\webpack\lib\NormalModuleFactory.js:117:11
at C:\....\node_modules\webpack\lib\NormalModuleFactory.js:628:24
at C:\....\node_modules\webpack\lib\NormalModuleFactory.js:782:8
at C:\....\node_modules\webpack\lib\NormalModuleFactory.js:902:5
@ ./node_modules/threebox-plugin/src/objects/loaders/FBXLoader.js 2:15-42
Alternatives
Some bundlers accept the option to ignore a reference to a module, but not all of them. I was guessing if you find an easy way to avoid this reference or make it conditional in the code. Any hint will be also greatly appreciated.
What can't you do right now?
I'm writing a wrapper around Inflate to use it as a sink for a WritableStream. However, unlike fflate's streaming implementation, finalizing a WritableStream is done by calling close(), which takes no parameters (no final chunk), and there's no way for a sink to know when a stream is finished until close() is called. I tried calling Inflate.push([], true) to signal to fflate to flush the stream but it throws an error (from the slc function at line 145):
RangeError: Invalid typed array length: -32768
Example script: https://gist.github.com/BitLooter/d507cf7c5b320e806fec06b22d2ccaa4
Tested with fflate 0.4.6 on Node 15.5.0. Note that while this is an implementation of a UnderlyingSink for a WritableStream (which Node doesn't support) it still runs in Node as it's not actually creating any stream objects, it's just calling methods on an object.
An optimal solution
I need a way to finalize an fflate stream without having any input data, to call from an UnderlyingSink's close() method.
(How) is this done by other libraries?
I have written a sink wrapper for Pako and solved this problem by pushing an empty chunk with the Z_FINISH flag, similar to what I attempted with fflate. I don't know if this is the best solution but that or a new method to finalize the stream without data would work.
When calling async functions (e.g. inflate
), a new web-worker is created.
This is problematic when calling it thousands of times, as it'll easily overload the browser with worker instantiations.
When the file-sizes are small it's quite problematic because the 70ms worker instantiation takes longer than the decompression.
The solution that comes to mind would be a worker-pool (whether in-library or by the user).
It may be worth noting this in the documentation for future readers.
Hi All,
I am checking this library showing extreme use of compression and decompression zip functionality. i want simple zip generate download option for number of files. please guide me for this.
Regards,
Sanny Soni (Web Developer)
The library works fine in dev mode in my vue app but will stop working when webpack minify the files to create the dist bundle. To make things working again I've added this line chainWebpack: config => config.optimization.minimize(false)
to my vue.config.js
file, it will disable files minification but will result in worst performances in case of large app.
This is the error that is logged in console after the production build of the app is created and the user try to create a zip file
Uncaught TypeError: Cannot read property 'length' of null
at A (chunk-vendors.64883a9e.js:6)
at chunk-vendors.64883a9e.js:6
at Worker.s.onerror (chunk-vendors.64883a9e.js:6)
A @ chunk-vendors.64883a9e.js:6
(anonymous) @ chunk-vendors.64883a9e.js:6
s.onerror @ chunk-vendors.64883a9e.js:6
error (async)
r @ chunk-vendors.64883a9e.js:6
X @ chunk-vendors.64883a9e.js:6
re @ chunk-vendors.64883a9e.js:6
ce @ chunk-vendors.64883a9e.js:6
h @ chunk-vendors.64883a9e.js:6
Ae @ chunk-vendors.64883a9e.js:6
(anonymous) @ write-zip.896bce5d.js:1
u @ chunk-vendors.64883a9e.js:8
(anonymous) @ chunk-vendors.64883a9e.js:8
(anonymous) @ chunk-vendors.64883a9e.js:8
a @ chunk-vendors.64883a9e.js:1
s @ chunk-vendors.64883a9e.js:1
Promise.then (async)
a @ chunk-vendors.64883a9e.js:1
s @ chunk-vendors.64883a9e.js:1
Promise.then (async)
a @ chunk-vendors.64883a9e.js:1
s @ chunk-vendors.64883a9e.js:1
Promise.then (async)
a @ chunk-vendors.64883a9e.js:1
s @ chunk-vendors.64883a9e.js:1
Promise.then (async)
a @ chunk-vendors.64883a9e.js:1
s @ chunk-vendors.64883a9e.js:1
Promise.then (async)
a @ chunk-vendors.64883a9e.js:1
s @ chunk-vendors.64883a9e.js:1
Promise.then (async)
a @ chunk-vendors.64883a9e.js:1
s @ chunk-vendors.64883a9e.js:1
Promise.then (async)
a @ chunk-vendors.64883a9e.js:1
s @ chunk-vendors.64883a9e.js:1
Promise.then (async)
a @ chunk-vendors.64883a9e.js:1
s @ chunk-vendors.64883a9e.js:1
Promise.then (async)
a @ chunk-vendors.64883a9e.js:1
s @ chunk-vendors.64883a9e.js:1
(anonymous) @ chunk-vendors.64883a9e.js:1
(anonymous) @ chunk-vendors.64883a9e.js:1
processFiles @ write-zip.896bce5d.js:1
(anonymous) @ write-zip.896bce5d.js:1
(anonymous) @ chunk-vendors.64883a9e.js:6
Ee @ chunk-vendors.64883a9e.js:6
Re @ chunk-vendors.64883a9e.js:6
t @ chunk-vendors.64883a9e.js:6
I tried using the latest version of fflate
, and I can't seem to get the zip()
method to work.
Example CSB
https://codesandbox.io/s/fflate-zip-files-yuu21
Steps to reproduce
zip()
is called.Additional context
I wrote a simple Promise
wrapper around the callback-based zip()
method so it is easier to make them work in async
/await
flows.
deo
is described in source as:
// deflate options (nice << 13) | chain
const deo = /*#__PURE__*/ new u32([65540, 131080, 131088, 131104, 262176, 1048704, 1048832, 2114560, 2117632]);
If nice
and chain
are understood to be the tunable nice_length
and max_chain
parameters as defined in zlib
, the constants look different from expectations. For example:
local const config configuration_table[10] = {
/* good lazy nice chain */
/* 0 */ {0, 0, 0, 0, deflate_stored}, /* store only */
/* 1 */ {4, 4, 8, 4, deflate_fast}, /* max speed, no lazy matches */
/* 2 */ {4, 5, 16, 8, deflate_fast},
/* 3 */ {4, 6, 32, 32, deflate_fast},
/* 4 */ {4, 4, 16, 16, deflate_slow}, /* lazy matches */
/* 5 */ {8, 16, 32, 32, deflate_slow},
/* 6 */ {8, 16, 128, 128, deflate_slow},
/* 7 */ {8, 32, 128, 256, deflate_slow},
/* 8 */ {32, 128, 258, 1024, deflate_slow},
/* 9 */ {32, 258, 258, 4096, deflate_slow}}; /* max compression */
Since deo
is indexed at level - 1
, it would seem that deo[2]
should be (32 << 13) | 32
(262176). The array would be unsorted in this case since the next level is (16 << 13) | 16
(131088)
How to reproduce
<script src="https://unpkg.com/fflate"></script>
<script src="https://cdn.jsdelivr.net/npm/[email protected]/dist/pako.js"></script>
<script>
let massiveFileBuf;
fetch('/f.html').then(
res => {
res.arrayBuffer().then(e => {
massiveFileBuf = new Uint8Array(e);
})
});
let resLen = 0
let startfflate = () => {
let ins = new fflate.Deflate({
level: 6
});
ins.ondata = function (data, final) {
resLen += data.length
}
ins.push(massiveFileBuf.slice(0), true);
console.log("full Deflate length",resLen);
resLen = 0;
let ins2 = new fflate.Deflate({
level: 6
});
ins2.ondata = function (data, final) {
resLen += data.length
}
let offset = 0;
let len = massiveFileBuf.length;
while (offset < len) {
ins2.push(massiveFileBuf.slice(offset, offset + 16), false);
offset += 16;
}
ins2.push(new Uint8Array(),true);
console.log("stream Deflate length",resLen);
}
let startpako = () => {
let ins = new pako.Deflate({
level: 6
});
ins.onData = function (data) {
resLen += data.length
}
ins.push(massiveFileBuf.slice(0), true);
console.log("full Deflate length",resLen);
resLen = 0;
let ins2 = new pako.Deflate({
level: 6
});
ins2.onData = function (data) {
resLen += data.length
}
let offset = 0;
let len = massiveFileBuf.length;
while (offset < len) {
ins2.push(massiveFileBuf.slice(offset, offset + 16), false);
offset += 16;
}
ins2.push(new Uint8Array(),true);
console.log("stream Deflate length",resLen);
}
</script>
The problem
As I tested, the console shows:
startfflate()
b.html:20 full Deflate length 755300
b.html:36 stream Deflate length 10362697
startpako()
b.html:46 full Deflate length 739044
b.html:62 stream Deflate length 739044
The fflate deflate stream result too big and even bigger than the origin data length which is 7832702.
So I wonder if the Deflate stream push function has some logic error.
How to reproduce
The problem
The compressed size is unrealistic, which makes me think that the file was not processed correctly. Instead an empty string or something similar was processed.
Browser: Latest Chrome
OS: Latest macOS
For TypeScript projects that have the strictNullChecks
TypeScript flag enabled
What can't you do right now?
It would be helpful to be able to do a partial (sync) flush during a streaming deflate, such that all the data passed to the deflater so far can be fully inflated at the other end, without ending the stream or fully resetting the compression state. This will enable implementation of packet-level compression in a binary stream protocol.
Example code
const deflate = new fflate.Zlib();
deflate.ondata = (data) => writeDataToStream(data);
deflate.push(packet1);
deflate.flush(); // <-- proposed new API
// At this point the flushed data is written to the stream,
// and the other side can fully decompress all packets sent so far.
// Later, more data can be sent, continuing the compressed stream
// without resetting the compression state.
deflate.push(packet2);
deflate.flush();
// Meanwhile the inflater is on the other side of a stream connection,
// receiving a stream of compressed data from the deflater.
const inflate = new fflate.Inflate();
inflate.ondata = handleReceivedPacket(data);
let receivedData;
while (!!(receivedData = getNextChunkFromStream())) {
deflate.push(receivedData);
// The `ondata` callback should be invoked once reaching each flush point
// (or earlier if there was a lot of data).
}
(How) is this done by other libraries?
The Node.js zlib
API supports this in two ways:
flush: zlib.constants.Z_SYNC_FLUSH
in options for zlib.createDeflate()
. Then it will auto-flush after every write.flush(zlib.constants.Z_SYNC_FLUSH)
whenever desired after writing some data to the deflate stream.Pako doesn't support this functionality as far as I can see. I don't know about other libraries.
In Webkit ( I tried both Safari macOS and Safari iOS ) an error is thrown when I call fflate.unzip
in a web worker :
ReferenceError: can't find variable: Worker
This is where it comes from :
Line 11 in 7471ec9
to be accurate I bundle with Webpack, I import unzip
like so :
import { unzip } from 'fflate'
and then I use it in this function :
unzip( new Uint8Array( buffer ), (err, unzipped) => {
if ( err ) reject( err );
resolve( unzipped.f1 );
});
Everything works like a charm in Chrome and Chrome for Android ( congrats for your work by the way ).
I tried with different files, big and small ( 1.5Mb to >300Mb ).
I believe this error occurs because Webkit doesn's support creating new web workers from inside web workers. I found a thread here with some vague references.
I need to unzip the file inside my own worker because I'm doing other expensive things after unzipping, it would not make sense to message this big buffer back and forth the main thread uselessly.
It's possible that I didn't understand how to use your library, and I should use another method to get this done. In that case can you give me a hint please ? My zipped files are compressed with fflate.zip
( in nodejs ).
possibly related: #11
The streaming API seems to lack support for stream backpressure. Without backpressure, it's not possible to ensure that we're not pushing data to the stream too quickly for it to handle. If we push data too quickly and the fflate stream can't keep up, then it's possible that the data will be sitting around in memory for longer than we want.
Stream backpressure helps us to detect this condition and to stop calling push()
until the backed up chunks have been processed.
The example toNativeStream
code used on the website doesn't support backpressure. Is it possible to add support for this?
Or, at least expose the necessary information (i.e. outstanding chunk count, and a way to call a user-provided callback when the outstanding chunk count falls below a threshold so we can start pushing again)?
Separately, it might be helpful to just offer a WHATWG TransformStream
interface to the user so they don't need to implement it themselves.
This project is awesome, it is the fast ,small and support ES module.
I want to know do you have any plans to support passworded zip files?
How to reproduce
The following assertion fails:
const fflate = require ( 'fflate' );
const char = String.fromCodePoint ( 55296 );
console.assert ( fflate.strFromU8 ( fflate.strToU8 ( char ) ) === char );
The problem
I'm not sure if this is a bug exactly, I think ideally any string should be able to go through both strToU8 and then strFromU8 and exit unchanged, but the problem with the string I picked is that it comprises of a single high surrogate character, which isn't actually valid UTF-16 as there must be a low surrogate character immediately after that in the string, but JS doesn't throw so you can actually have strings like that.
In my case, I need to replace a file in a zip file
For example, in my zip file a.zip, it has 2 files, like a.txt, b.txt.
I only want to replace a.txt's content, or remove a.txt, and I don't want to unzip all files in memory
And I need to use as little memory as possible,cause i have a big zip file, and the memory is not cheap in my case
By now, I zip all files in 4 s with 700m,and I have no way to replace serveral files with little memory and time.
Can you help me with some ideas?
What can't you do right now?
Aes encrypt for zip.
An optimal solution
Use sjcl to do it.
Could I add a zip aes (en/de)crypt module?
And if I could add that module, could you give me some suggestions?
What can't you do right now?
Zstandard compresses amazingly well, for my use case up to 4x better than what I can get from gzip, and it decompresses perhaps even more amazingly well.
For my use case I kinda have to use Zstandard given its characteristics, but there doesn't seem to be a great port of Zstandard for the browser, plus I'm already using fflate
which I find well done and pretty fast, but it just doesn't support Zstandard.
An optimal solution
Optimally fflate
would add support for compressing and decompressing (although just decompressing would be enough for me, and I'd guess for most other people too) using Zstandard with a high quality and fast implementation.
(How) is this done by other libraries?
node-zstd
: it uses native bindings, a no-no for the web.
node-zstandard
: it just spawns zstd as a child process.
zstd-codec
: it's a 1 year out of date emscripten compiled version of zstd, which weighs almost 1MB min+gzipped and can't be tree-shaken.
zstddec
: this seems the best port to webassembly, it only supports decoding and weighs ~17kb, but it has the following pretty concerning warning in its readme:
Limitations: The decoder may fail with the error wasm function signature contains illegal type when the uncompressedSize is not known in advance and given to the decode() method. This is presumably a bug in the WASM bindings, which I am not yet sure how to fix.
Basically there currently isn't a great option for decoding zstd files on the web and I'd love to see one from fflate
.
Hi :)
I recently started using this library in one of mine, basically as part of the build I statically compress a WASM file (level: 9) and then on load inflate it. This has been tested quite wide under browsers and Node.js (.cjs and .mjs) and it is doing quite well with only a slight overhead, but quite nice overall savings in the bundles (even when served with compression)
However, just ran into an issue logged by one of my users and don't quite know a way forward to improve matters and solve this issue as it stands.
So this (and it may also exist in mixed Node/browser frameworks as next.js - untested) basically means that the bundle serves from a node environment (which ignores the browser field in package.json), but executes in another environment, which actually does need that override and/or it doesn't exist. (Not quite sure which option comes into play here, really not quite familiar with fastlify)
I actually only ever use unzlibSync
(See https://github.com/polkadot-js/wasm/blob/master/packages/wasm-crypto-wasm/src/data.ts#L8), so in a perfect world would have avoided anything worker-related completely. However, in this case do get tripped up.
Any suggestions to work around this and or solve this? Preferable on this level?
Would really like to solve the linked issue, but don't quite see any options - I guess if the parts were split into their own files (or even just sync/async), could have possibly done a direct import, avoiding workers completely. (Well, then we have tree-shaking issues since atm an export map is not required).
Uploading multiple files from a worker, creates a memory leak and eventually throws the following error:
"Cannot perform Construct on a detached ArrayBuffer"
Looking at the memory snapshot between two runs it seems like the created array buffers are not being cleaned after running the compress method
Would be awesome to add support for reading&writing the "extra fields" from the specifications: https://datatracker.ietf.org/doc/html/rfc1952#page-8
2.2. File format
A gzip file consists of a series of "members" (compressed data
sets). The format of each member is specified in the following
section. The members simply appear one after another in the file,
with no additional information before, between, or after them.
2.3. Member format
Each member has the following structure:
+---+---+---+---+---+---+---+---+---+---+
|ID1|ID2|CM |FLG| MTIME |XFL|OS | (more-->)
+---+---+---+---+---+---+---+---+---+---+
(if FLG.FEXTRA set)
+---+---+=================================+
| XLEN |...XLEN bytes of "extra field"...| (more-->)
+---+---+=================================+
(if FLG.FNAME set)
+=========================================+
|...original file name, zero-terminated...| (more-->)
+=========================================+
(if FLG.FCOMMENT set)
+===================================+
|...file comment, zero-terminated...| (more-->)
+===================================+
(if FLG.FHCRC set)
+---+---+
| CRC16 |
+---+---+
+=======================+
|...compressed blocks...| (more-->)
+=======================+
0 1 2 3 4 5 6 7
+---+---+---+---+---+---+---+---+
| CRC32 | ISIZE |
+---+---+---+---+---+---+---+---+
2.3.1.1. Extra field
If the FLG.FEXTRA bit is set, an "extra field" is present in
the header, with total length XLEN bytes. It consists of a
series of subfields, each of the form:
+---+---+---+---+==================================+
|SI1|SI2| LEN |... LEN bytes of subfield data ...|
+---+---+---+---+==================================+
SI1 and SI2 provide a subfield ID, typically two ASCII letters
with some mnemonic value. Jean-Loup Gailly
<[email protected]> is maintaining a registry of subfield
IDs; please send him any subfield ID you wish to use. Subfield
IDs with SI2 = 0 are reserved for future use. The following
IDs are currently defined:
Thanks in advance!
Hello, I think adding http://lz4.github.io/lz4/ support could be very useful.
Even more with stream feature if possible
The problem
I noticed that there are several places where you throw
a string. It's considered best practice to only throw Error
objects. When listening to an ondata
event, it's really unexpected to have a variable called err
that is a string instead of an Error
object.
I am seeing a few issues with Gzip/Gunzip on Deno.
One is, compressing then decompressing modest size text files, give corrupt and much smaller uncompressed version.
Another is compressing a modest sized binary file is generating an invalid compressed file format, and when attempting to decompress get an Invalid gzip data exception.
I wrote the following test to demonstrate the problem:
import * as fflate from 'https://cdn.skypack.dev/fflate';
async function zpipe(reader: Deno.Reader, stream: any) {
let total = 0;
async function push(p: Uint8Array, isLast?: boolean) {
console.log('push', p.byteLength);
debugger;
await stream.push(p, isLast);
total += p.byteLength;
}
let prevBlock;
for await (const block of Deno.iter(reader)) {
if (prevBlock) await push(prevBlock);
prevBlock = block;
}
if (prevBlock) await push(prevBlock, true);
console.log(`pushed ${total} bytes`);
}
function zip(from: string, to: string, options = {}) {
let total = 0;
return new Promise<void>(async (resolve, reject) => {
const hFrom = await Deno.open(from, { read: true });
const hTo = await Deno.open(to, { write: true, create: true, truncate: true });
const zipper: any = new fflate.Gzip({ level: 9 }, async (chunk: Uint8Array, isLast: boolean) => {
console.log('zip write chunk', chunk.byteLength);
await hTo.write(chunk);
total += chunk.byteLength;
if (isLast) {
console.log(`zip close dest file, ${total} bytes`);
hTo.close();
resolve();
}
});
await zpipe(hFrom, zipper);
console.log('zip close source file');
hFrom.close();
});
}
function unzip(from: string, to: string) {
let total = 0;
return new Promise<void>(async (resolve, reject) => {
const hFrom = await Deno.open(from, { read: true });
const hTo = await Deno.open(to, { write: true, create: true, truncate: true });
const unzipper: any = new fflate.Gunzip();
unzipper.ondata = async (chunk: Uint8Array, isLast: boolean) => {
console.log('unzip write chunk', chunk.byteLength);
await hTo.write(chunk);
total += chunk.length;
if (isLast) {
console.log(`unzip close dest file, ${total} bytes`);
hTo.close();
resolve();
}
};
await zpipe(hFrom, unzipper);
console.log('unzip close source file');
hFrom.close();
});
}
const fn = Deno.args[0];
await zip(fn, `${fn}.gz`);
await unzip(`${fn}.gz`, `${fn}.unzipped`);
As a test, I downloaded fflate.js and compressed and decompressed that using the code above:
deno run --allow-all gzip.ts fflate.js
The resulting file sizes are:
-a---- 20/03/2021 15:43 54748 fflate.js
-a---- 21/03/2021 00:57 14322 fflate.js.gz
-a---- 21/03/2021 00:57 16384 fflate.js.unzipped
For the binary file test, I generated a 32kb binary random file using dd
dd if=/dev/random of=LARGE_FILE ibs=1k count=32
Then compress it with the above code:
deno run --allow-all gzip.ts LARGE_FILE
This throws an error on the unzip:
error: Uncaught (in promise) invalid gzip data
And file reports a strange size on the compressed file:
LARGE_DATA.gz: gzip compressed data, last modified: Sun Mar 21 01:01:11 2021, max compression, from Unix, original size modulo 2^32 100822718
Thanks for the library! Been testing it out and ran into an interesting error:
const { strFromU8, zlibSync, strToU8, unzlibSync } = require('fflate')
const str = 'hi'
let compressed, decompressed
// This works
compressed = zlibSync(strToU8(str))
console.log(compressed)
decompressed = strFromU8(unzlibSync(compressed))
console.log(decompressed)
// This as well
compressed = strFromU8(zlibSync(strToU8(str)), true)
console.log(compressed)
decompressed = strFromU8(unzlibSync(strToU8(compressed, true)))
console.log(decompressed)
// This does not
compressed = strFromU8(zlibSync(strToU8(str)))
console.log(compressed)
decompressed = strFromU8(unzlibSync(strToU8(compressed)))
console.log(decompressed)
The output for version 0.4.1 in node.js is:
⟩ node s.js
Uint8Array(13) [
120, 156, 1, 2, 0,
253, 255, 104, 105, 1,
59, 0, 210
]
hi
x���ýÿhi�;Ò
hi
x�����hi�;�
/home/macobo/yyy/node_modules/fflate/lib/index.js:887
throw 'invalid zlib data';
^
invalid zlib data
(Use `node --trace-uncaught ...` to show where the exception was thrown)
The same seems to occur for other compression functions as well - tested gzipSync, compressSync. Could there be a problem with strFromU8
?
It would be nice to have the ability to create shared context/sliding window
let inflate = new Inflate({ chunkSize: 128 * 1024 })
inflate.push(chunk)
inflate.push(chunk, Z_SYNC_FLUSH)
console.log(inflate.result.toString())
How to reproduce
The problem
Hi I'm trying to gzip some files in a .then()
promise chain so I tried using gzipAsync()
but received the following error:
Error: async operations unsupported - update to Node 12+ (or Node 10-11 with the --experimental-worker CLI flag)
at Immediate.<anonymous> (file:///home/today/_/work/haptic/node_modules/fflate/esm/index.mjs:34:42)
at processImmediate (internal/timers.js:456:21)
I'm running Node v14.2.0. The file is ESM via "type": "module"
in package.json. The file imports fflate like this:
import esbuild from 'esbuild';
import { gzip, gzipSync } from 'fflate';
import { readFile, writeFile } from 'fs/promises';
// ...
Here's the code that throws on L47:
If I replace L47 with res(gzipSync(readData, { consume: true, level: 9 }));
then it's fine. It's also fast enough that I'm not reporting an issue with Sync vs Async, just that the error message might be wrong if it's telling people to update to 12+.
Thanks!
Repro steps
npm install
node index.js
Output
~/: node index.js
~/node_modules/fflate/lib/index.js:272
throw 'unexpected EOF';
I've encountered a few corrupted files after using gzip with fflate, both the async and sync versions. Results appear to be deterministic, always occuring on the same files on different machines. I've made sure that the issue is somewhere in fflate. (removed all other variables and side-by-side with pako. Pako did not produce any corruption)
Issue only occurs on a small subset of files, less then 10% of what we use. It's always rather large files, in the >200MB range. I tried changing the compression level, and this gave some good initial results. However, it appears that another compression level just causes corruption in completely different files.
I've currently reverted back to pako, but would like to continue using fflate once this is resolved. If you want, I can send you example files that get corrupted to help with debugging this. (These are large medical files stripped of identifiable data, yet still to sensitive to just upload publically ;) )
It would be great to have some sort of progress callback which could give an estimated percentage of the completed work. Am willing to help out with this feature if you can point me in the right direction. :)
Hey! Great library!
I am using it in the browser, and it would be great if the async versions would support promises as well instead of just callbacks.
Ideally it would look something like this:
const buffer = await fflate.zip(
{
'index.html': fflate.strToU8(str),
// other files...
},
{ level: 4 }
);
It looks like synchronous code but it doesn't block the main thread.
It makes sense to support promises, newer versions of node are doing so with stuff like fs/promises
and timers/promises
Supporting creating and extracting ZIP files would be very useful, especially for real-world use cases like zipping multiple uploads into one on the client-side and then uploading to a pre-signed S3 URL.
Not sure if this was intended, but I've been doing:
import * as fflate from 'https://unpkg.com/fflate/esm/index.mjs';
As of 0.5.3, this fails, because the following code at the top of index.mjs
:
try {
Worker = /*#__PURE__*/ require('worker_threads').Worker;
}
catch (e) {
}
has moved to node-worker.js
and been replaced with:
import wk from './node-worker.js';
which doesn't work in a browser because node-worker.js
doesn't define a default export, or any exports at all.
How to reproduce
Attempt to use this library in an environment that does not provide the setTimeout
API. In my case, that is QML.
The problem
QML does not provide the setTimeout
API. I can fake it, but from looking at the code it seems it might be possible to stop using setTimeout
completely.
What can't you do right now?
As documented in the changelog, zip files cannot contain files greater than 4GB. It would be great if we could create zip files containing larger files, since it looks like the zip64 format can support that use case.
An optimal solution
Ideally, the current streaming interface would automatically generate valid zip output when adding larger than 4GB files. It may also be necessary for the user to pass an option indicating that we need a zip64 header when adding a large file; that would be fine too.
As an interim solution, instead of adding full zip64 support, it would be nice if ondata
returned an error when trying to write too large of a file instead of silently generating a corrupt .zip file. Currently, it looks like the generated .zip is the right size, but when unzipped it outputs truncated files. I suspect the size field just wraps around at 4GB; could you detect that and return an error?
What can't you do right now?
Recently I've been trying to generate USDZ files in JavaScript: mrdoob/three.js#21245
The USDZ file is essentially an zip file with 0 compression that contains a 3d file and textures.
So far fflate has been great for creating the file. However, yesterday I learned that in order to produce valid USDZ files I need to make sure that the files are aligned to 64 bytes. Currently my files are invalid:
From the spec page:
I'm aware this is quite an uncommon use case, but would you be interested in adding support for this?
Some observations that I noted:
inflate
fails with (null, null)
inflateSync
fails with EOF:This is probably the most important problem since one would expect either data or an error.
decompress
works fine:deflateSync
: incorrect header checkIt appears as though Pako has a different idea of what constitutes inflate
/ deflate
i.e.
gzipSync
: seems to correspond to pako.inflate
I noted that the builtin CompressionStream
seems to correlate with what Pako does i.e. 'inflate'
algorithm for Pako and CompressionStream seem to be the same.
zlib
: same as gzip
zip
: fails, same as deflateI've no idea what's going on here (and the issues observed may be with Pako), but I hope the above is useful feedback.
How to reproduce
This zip seems to reproduce the issue: https://sethealth-customers.b-cdn.net/repo.zip
Try to parse the ZIP file with a single huge chunk (just trying to reproducee the issue):
import {Unzip, UnzipInflate} from 'fflate';
import {readFileSync} from 'fs';
const readFile = (file) => {
return new Promise((resolve, reject) => {
const chunks = [];
file.ondata = (err, chunk, final) => {
if (err) {
reject(reject);
}
if (chunk) {
chunks.push(chunk);
}
if (final) {
resolve(concatBuffers(chunks));
}
};
file.start();
});
}
const concatBuffers = (arrays) => {
const totalLength = arrays.reduce(
(acc, value) => acc + value.byteLength,
0
);
const output = new Uint8Array(totalLength);
let offset = 0;
for (let array of arrays) {
output.set(array, offset);
offset += array.byteLength;
}
return output;
};
const chunk = new Uint8Array(readFileSync("repo.zip"));
const promises = [];
const unzipper = new Unzip((handler) => {
promises.push((async () => {
return {
name: handler.name,
stream: await readFile(handler),
}
})());
});
unzipper.register(UnzipInflate);
unzipper.push(chunk, false);
unzipper.push(new Uint8Array(), true);
const output = await Promise.all(promises);
console.log(output);
^ This code will crash with "Invalid zip data".
The problem
fflate seems to unzip correctly this file when using unzipSync()
, but fails in stream mode:
import {readFileSync} from 'fs';
import {unzipSync} from 'fflate';
const chunk = new Uint8Array(readFileSync("repo.zip"));
const output = unzipSync(chunk);
console.log(output);
compressing in ios is crashing. I didn't check in android.
the code I used is
const data = zipSync(tozip, { level: 0, mem: 0 });
saveAs(new Blob([data as Uint8Array]), `${this.title}.zip`);
the result zip file made with pc is about 200~300mb and including about 80 jpg photos.
Thank you, I always appreciate your work.
How to reproduce
https://codepen.io/manucorporat/pen/PopXJgb?editors=1010
Notice the:
const step = 5141811;
With this specific value, the ungzip breaks with Error: invalid length/literal
, but works with other values, higher and lower.
This variables allows to rechunk the file into smaller pieces.
This code is an example to minimally reproduce the issue, in production we got this error with random setups.
The problem
Gunzip streaming API sometimes errors.
What can't you do right now?
Use the async methods as promises without using promisify (and for the browser, needing to bundle or otherwise add a promisfy utility).
An optimal solution
Be able to use the async methods as promises.
(How) is this done by other libraries?
JSZip appears to support promises, but not the other libraries.
Thanks!
Hi,
thank you for your comment at Stuk/jszip#721 (comment)
I am trying to review fflate if it would meet my requirements. However, the documentation is not very good and I cannot get my compressed zip file unzipped. Can you help out here?
My code from jszip
//zippedData is binary String
let foundXml = "";
try {
var JSZip = require("jszip");
var zip = new JSZip();
const zipped = await zip.loadAsync(zippedData);
foundXml = await zipped.file("data.xml").async("string");
} catch(err) {
console.log(err);
}
What i tried in fflate
try {
let buffer = fflate.strToU8(zippedData, true);
let unzipped = fflate.unzipSync(buffer);
console.log("unzipped", unzipped); //-> this returns {"data.xml": []}
console.log("unzipped", unzipped["data.xml"]); //-> this returns []
} catch(err) {
console.log(err);
}
So I get the file unzipped but the data.xml inside is empty.
Hello, when I try to unzip a zip made with zip
I have this error
RangeError: Invalid typed array length: -48783
Codesandbox with reproduction:
https://codesandbox.io/s/friendly-kare-dgcls
This happens on Chrome, does not seems to throw an error on Safari but doesn't work either
Might be related with #21 but I don't use any stream
Also I'm using v0.4.7
Thanks in advance :)
yarn v1.22.10; node v14.15.0; webpack v5.13.0
Hello, I did not have a problem with previous versions of the fflate v4.x but starting from v5.0:
// client side (browser) webworker
import { gunzipSync } from 'fflate';
...
webpack 5.13.0 compiled successfully in 10038 ms
internal/process/esm_loader.js:74
internalBinding('errors').triggerUncaughtException(
^
Error [ERR_REQUIRE_ESM]: Must use import to load ES Module: /../node_modules/fflate/esm/index.mjs
at Module.load (internal/modules/cjs/loader.js:926:11)
at Function.Module._load (internal/modules/cjs/loader.js:769:14)
at Module.require (internal/modules/cjs/loader.js:952:19)
at require (internal/modules/cjs/helpers.js:88:18)
at Object.fflate (/../build/bundle.js:1069:18)
at webpack_require (/../build/bundle.js:1332:41)
at eval (webpack://mpd-eds/./src/client/components/decompressFileContent.js?:17:64)
at Object../src/client/components/decompressFileContent.js (/../build/bundle.js:599:1)
at webpack_require (/../build/bundle.js:1332:41)
at eval (webpack://mpd-eds/./src/client/actions/initState.js?:57:91) {
code: 'ERR_REQUIRE_ESM'
}
[nodemon] app crashed - waiting for file changes before starting...
...
Test:
https://codesandbox.io/s/sweet-ritchie-g49rw?file=/src/index.js
Encoding - zlib deflate
Actual: inifinity loop inside decoding phase
Expected: faster that pako
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.