google / gipfeli Goto Github PK
View Code? Open in Web Editor NEWLicense: Other
License: Other
Gipfeli, a high-speed compression library. Introduction ============ Gipfeli is a high-speed compression/decompression library aiming at slightly higher compression ratios (around 30 % less bytes produced for text) than other high-speed compression libraries. On a single core of a Core i7 processor in 64-bit mode, gipfeli compresses at about 180 MB/s or more and decompresses at about 300 MB/s, typically 5x faster than zlib, but does not quite achieve its compression density. Gipfeli is written in C++, with no assembler code. Contact ======= Gipfeli is distributed through GitHub. For the latest version, a bug tracker, and other information, see https://github.com/google/gipfeli
I'm improving the random-data test case in Squash to try buffers of random data up to 8 MiB, and I've noticed that Gipfeli starts to fail around a couple hundred KiB due to the decompressed data not matching the input. An example of problematic input is at http://code.coeusgroup.com/gipfeli-bug (357 KiB)
To verify that the issue was in Gipfeli and not Squash, I threw together this test:
#include <stdio.h>
#include <assert.h>
#include "gipfeli.h"
#include "stubs-internal.h"
int main(int argc, char *argv[]) {
FILE* input = fopen (argv[1], "rb");
assert (input != NULL);
int r = fseek (input, 0, SEEK_END);
assert (r == 0);
size_t uncompressed_size;
{
long p = ftell (input);
assert (p > 0);
uncompressed_size = (size_t) p;
}
fseek (input, SEEK_SET, 0);
uint8_t* uncompressed = (uint8_t*) malloc (uncompressed_size);
assert (uncompressed != NULL);
fread (uncompressed, 1, uncompressed_size, input);
fclose (input);
input = NULL;
util::compression::Compressor* compressor =
util::compression::NewGipfeliCompressor();
size_t compressed_size = compressor->MaxCompressedLength (uncompressed_size);
uint8_t* compressed = (uint8_t*) malloc (compressed_size);
{
util::compression::UncheckedByteArraySink sink((char*) compressed);
util::compression::ByteArraySource source((const char*) uncompressed, uncompressed_size);
compressed_size = compressor->CompressStream (&source, &sink);
fprintf (stderr, "Compressed %p[%zu] -> %p[%zu]\n", uncompressed, uncompressed_size, compressed, compressed_size);
}
{
std::string compressed_str((const char*) compressed, compressed_size);
size_t decompressed_size;
if (!compressor->GetUncompressedLength (compressed_str, &decompressed_size)) {
abort ();
}
assert (decompressed_size == uncompressed_size);
uint8_t* decompressed = (uint8_t*) malloc (decompressed_size);
assert (decompressed != NULL);
util::compression::UncheckedByteArraySink sink((char*) decompressed);
util::compression::ByteArraySource source((const char*) compressed, compressed_size);
if (compressor->UncompressStream (&source, &sink)) {
fprintf (stderr, "Decompressed %p[%zu] -> %p[%zu]\n", compressed, compressed_size, decompressed, decompressed_size);
} else {
fprintf (stderr, "Failed to decompress.\n");
abort ();
}
int res = memcmp (uncompressed, decompressed, uncompressed_size);
assert (res == 0);
free (decompressed);
}
delete compressor;
free (compressed);
free (uncompressed);
return 0;
}
Just feed it the name of the uncompressed file as the first argument, it should trigger the issue. It works for most files, but seems not to like larger blocks of random data.
Snappy provides a (pure) C interface โ do you plan to provide one as well, perhaps with a similar API?
__powerpc64__works for both.
Test case in commit in google/snappy#67
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.