lzma-native

Node.js interface to the native liblzma compression library (.xz file format, among others)

This package provides interfaces for compression and decompression of .xz (and legacy .lzma) files, both stream-based and string-based.

Example usage

Installation

Simply install lzma-native via npm:

$ npm install --save lzma-native

For streams

If you don’t have any fancy requirements, using this library is quite simple:

var lzma = require('lzma-native');
var compressor = lzma.createCompressor();

process.stdin.pipe(compressor).pipe(process.stdout);

For decompression, you can simply use lzma.createDecompressor().

Both functions return a stream where you can pipe your input in and read your (de)compressed output from.

For simple strings/Buffers

If you want your input/output to be Buffers (strings will be accepted as input), this even gets a little simpler:

lzma.compress('Banana', function(result) {
    console.log(result); // <Buffer fd 37 7a 58 5a 00 00 01 69 22 de 36 02 00 21 ...>
});

Again, replace lzma.compress with lzma.decompress and you’ll get the inverse transformation.

If you have Q available, lzma.compress and lzma.decompress will return Q promises and you don’t need any kind of callback.

API

Compatibility implementations

Apart from the API described here, lzma-native implements the APIs of the following other LZMA libraries so you can use it nearly as a drop-in replacement:

• node-xz via lzma.Compressor and lzma.Decompressor
• LZMA-JS via lzma.LZMA().compress and lzma.LZMA().decompress, though without actual support for progress functions and returning Buffer objects instead of integer arrays. (This produces output in the .lzma file format, not the .xz format!)

Reference

Encoding strings and Buffer objects

• compress() – Compress strings and Buffers
• decompress() – Decompress strings and Buffers
• LZMA().compress() (LZMA-JS compatibility)
• LZMA().decompress() (LZMA-JS compatibility)

Creating streams for encoding

• createCompressor() – Compress streams
• createDecompressor() – Decompress streams
• createStream() – (De-)Compression with advanced options
• Compressor() (node-xz compatibility)
• Decompressor() (node-xz compatibility)

Functions

• isXZ() – Test Buffer for .xz file format
• crc32() – Calculate CRC32 checksum
• checkSize() – Return required size for specific checksum type
• easyDecoderMemusage() – Expected memory usage
• easyEncoderMemusage() – Expected memory usage
• rawDecoderMemusage() – Expected memory usage
• rawEncoderMemusage() – Expected memory usage
• versionString() – Native library version string
• versionNumber() – Native library numerical version identifier

Encoding strings and Buffer objects

lzma.compress(), lzma.decompress()

lzma.compress(string, [opt, ]on_finish) lzma.decompress(string, [opt, ]on_finish)

If Q is available, a promise will be returned.

Example code:

lzma.compress('Bananas', 9, function(result) {
	lzma.decompress(result, function(decompressedResult) {
		assert.equal(decompressedResult.toString(), 'Bananas');
	});
});

Example code for Q promises:

lzma.compress('Bananas', 9).then(function(result) {
	return lzma.decompress(result);
}).then(function(decompressedResult) {
	assert.equal(decompressedResult.toString(), 'Bananas');
}).done();

lzma.LZMA().compress(), lzma.LZMA().decompress()

lzma.LZMA().compress(string, mode, on_finish[, on_progress]) lzma.LZMA().decompress(string, on_finish[, on_progress])

(Compatibility; See LZMA-JS for the original specs.)

Note that the result of compression is in the older LZMA1 format (.lzma files). This is different from the more universally used LZMA2 format (.xz files) and you will have to take care of possible compatibility issues with systems expecting .xz files.

If Q is available, a promise will be returned.

Example code:

lzma.LZMA().compress('Bananas', 4, function(result) {
	lzma.LZMA().decompress(result, function(decompressedResult) {
		assert.equal(decompressedResult.toString(), 'Bananas');
	});
});

For an example using Q promises, see compress().

Creating streams for encoding

lzma.createCompressor(), lzma.createDecompressor()

lzma.createCompressor([options]) lzma.createDecompressor([options])

Return a duplex stream, i.e. a both readable and writable stream. Input will be read, (de)compressed and written out. You can use this to pipe input through this stream, i.e. to mimick the xz command line util, you can write:

var compressor = lzma.createCompressor();

process.stdin.pipe(compressor).pipe(process.stdout);

The output of compression will be in LZMA2 format (.xz files), while decompression will accept either format via automatic detection.

lzma.Compressor(), lzma.Decompressor()

lzma.Compressor([preset], [options]) lzma.Decompressor([options])

(Compatibility; See node-xz for the original specs.)

These methods handle the .xz file format.

Return a duplex stream, i.e. a both readable and writable stream. Input will be read, (de)compressed and written out. You can use this to pipe input through this stream, i.e. to mimick the xz command line util, you can write:

var compressor = lzma.Compressor();

process.stdin.pipe(compressor).pipe(process.stdout);

lzma.createStream()

lzma.createStream(coder, options)

Return a duplex stream for (de-)compression. You can use this to pipe input through this stream.

The available coders are (the most interesting ones first):

• easyEncoder Standard LZMA2 (.xz file format) encoder. Supports options.preset and options.check options.
• autoDecoder Standard LZMA1/2 (both .xz and .lzma) decoder with auto detection of file format. Supports options.memlimit and options.flags options.
• aloneEncoder Encoder which only uses the legacy .lzma format. Supports the whole range of LZMA options.

Less likely to be of interest to you, but also available:

• aloneDecoder Decoder which only uses the legacy .lzma format. Supports the options.memlimit option.
• rawEncoder Custom encoder corresponding to lzma_raw_encoder (See the native library docs for details). Supports the options.filters option.
• rawDecoder Custom decoder corresponding to lzma_raw_decoder (See the native library docs for details). Supports the options.filters option.
• streamEncoder Custom encoder corresponding to lzma_stream_encoder (See the native library docs for details). Supports options.filters and options.check options.
• streamDecoder Custom decoder corresponding to lzma_stream_decoder (See the native library docs for details). Supports options.memlimit and options.flags options.

Options

options.filters can, if the coder supports it, be an array of filter objects, each with the following properties:

• .id Any of lzma.FILTERS_MAX, lzma.FILTER_ARM, lzma.FILTER_ARMTHUMB, lzma.FILTER_IA64, lzma.FILTER_POWERPC, lzma.FILTER_SPARC, lzma.FILTER_X86 or lzma.FILTER_DELTA, lzma.FILTER_LZMA1, lzma.FILTER_LZMA2

The delta filter supports the additional option .dist for a distance between bytes (see the xz(1) manpage).

The LZMA filter supports the additional options .dict_size, .lp, .lc, pb, .mode, nice_len, .mf, .depth and .preset. See the xz(1) manpage for meaning of these parameters and additional information.

Functions

lzma.isXZ()

lzma.isXZ(input)

Tells whether an input buffer is an XZ file (.xz, LZMA2 format) using the file format’s magic number. This is not a complete test, i.e. the data following the file header may still be invalid in some way.

Example usage:

lzma.isXZ(fs.readFileSync('test/hamlet.txt.xz')); // => true
lzma.isXZ(fs.readFileSync('test/hamlet.txt.lzma')); // => false
lzma.isXZ('Banana'); // => false

(The magic number of XZ files is hex fd 37 7a 58 5a 00 at position 0.)

lzma.crc32()

lzma.crc32(input[, encoding[, previous]])

Compute the CRC32 checksum of a Buffer or string.

Example usage:

lzma.crc32('Banana') // => 69690105

lzma.checkSize()

lzma.checkSize(check)

Return the byte size of a check sum.

Example usage:

lzma.checkSize(lzma.CHECK_SHA256) // => 16
lzma.checkSize(lzma.CHECK_CRC32)  // => 4

lzma.easyDecoderMemusage()

lzma.easyDecoderMemusage(preset)

Returns the approximate memory usage when decoding using easyDecoder for a given preset.

Example usage:

lzma.easyDecoderMemusage(6) // => 8454192

lzma.easyEncoderMemusage()

lzma.easyEncoderMemusage(preset)

Returns the approximate memory usage when encoding using easyEncoder for a given preset.

Example usage:

lzma.easyEncoderMemusage(6) // => 97620499

lzma.rawDecoderMemusage()

lzma.rawDecoderMemusage(filters)

Returns the approximate memory usage when decoding using rawDecoder for a given filter list.

lzma.rawEncoderMemusage()

lzma.rawEncoderMemusage(filters)

Returns the approximate memory usage when encoding using rawEncoder for a given filter list.

lzma.versionString()

lzma.versionString()

Returns the version of the underlying C library.

Example usage:

lzma.versionString() // => '5.2.1'

lzma.versionNumber()

lzma.versionNumber()

Returns the version of the underlying C library.

Example usage:

lzma.versionNumber() // => 50020012

Installation

This package includes the native C library, so there is no need to install it separately.

Licensing

The original C library package contains code under various licenses, with its core (liblzma) being public domain. See its contents for details. This wrapper is licensed under the LGPL 3 or any later version of the LGPL.

Related projects

Other implementations of the LZMA algorithms for node.js and/or web clients include:

• lzma-purejs
• LZMA-JS
• node-xz (native)
• node-liblzma (native)

Note that LZMA has been designed to have much faster decompression than compression, which is something you may want to take into account when choosing an compression algorithm for large files. Almost always, LZMA achieves higher compression ratios than other algorithms, though.

Acknowledgements

This project is financially supported by the Tradity project.