In decodedLen(),

    if uint64(int(v)) != v {
        return 0, 0, errors.New("snappy: decoded block is too large")
    }

However, the snappy format description (and the C and C++ implementations) limit this varint to 32-bits:

https://github.com/google/snappy/blob/master/format_description.txt

The stream starts with the uncompressed length (up to a maximum of 2^32 - 1),
stored as a little-endian varint.

This check should instead use

    if uint64(uint32(v)) != v {
        return 0, 0, errors.New("snappy: decoded block is too large")
    }

I can file a PR here or a CL if this project is using gerrit if you'd like.

This bug dates from the Go 1.1 integer size change.

Hi,

one of our projects is experiencing Travis CI build failures because of golang/snappy.

The build job on golang 1.4 is erroring with a following message:

# github.com/golang/snappy
../../golang/snappy/encode_amd64.s:208 redeclaration of end
../../golang/snappy/encode_amd64.s:564 redeclaration of end

Full log is located here, with the error message starting on line 350 - https://gist.github.com/serejja/e4593342cb002370c3c95326a6cce1c0

Just to provide a bit more info (which is in that log anyway):

1. go version go1.4 linux/amd64
2. Ubuntu 12.04.5 LTS, 3.13.0-29-generic
3. gcc (Ubuntu/Linaro 4.6.3-1ubuntu5) 4.6.3

Thanks!

Until now we've been importing snappy using code.google.com/p/snappy-go/snappy but recently this has caused this issue:

package code.google.com/p/snappy-go/snappy: unable to detect version control system for code.google.com/ path

By changing the import to github.com/golang/snappy/snappy, we seem to have resolved the issue.

Is this the recommended way to reference this library?

The signature is:

func Encode(dst, src []byte) ([]byte, error)

The current implementation only ever returns nil for the errors.

Furthermore, how could this function ever return an error? Aren't any possible 
values of dst and src valid?

For comparison, encoding/hex has

func Encode(dst, src []byte) int
func Decode(dst, src []byte) (int, error)

Original issue reported on code.google.com by [email protected] on 1 Oct 2014 at 11:22

I'm sorry if this is flat-out wrong or just too unimportant to fix; I'm new 
here.

Encode allocates MaxEncodedLen(len(src)) bytes for dst. However, the (very 
well-explained) scenario motivating the formula in MaxEncodedLen(srcLen) 
involves an encoder emitting 5-byte-long copy elements each copying four bytes. 
But snappy-go can't encounter that scenario because it emits at most 
3-byte-long copy elements, each copying at least four bytes, regardless of 
block length. 

Given the restrictions on what snappy-go emits, the worst case I can see is a 
3-byte-long copy element that copies four bytes, followed by a 2-byte-long 
literal element with length 61, followed by the literal bytes, repeated for the 
whole block: then it takes 66 bytes to encode every 65 bytes. Add three bytes 
for the length, and 64K encodes to 65536*(66/65)+3=66547 bytes, 10393 fewer 
than the 76940 bytes actually allocated. Maybe someone can find a worse case.

If snappy-go did, in the future, emit 5-byte copy elements, presumably it still 
wouldn't do so in a 64K block, because offsets within a 64K block always fit in 
two bytes. 

(If MaxEncodedLen is meant to represent the worst possible blowup with the 
Snappy encoding, rather than the worst possible blowup with snappy-go or other 
currently-existing encoders, note that the encoding appears to allow 
5-byte-long copy elements to only copy one byte, so the absolute most 
pessimistic upper bound may be len(src)*5 plus space for the preamble.)

Original issue reported on code.google.com by [email protected] on 6 Aug 2014 at 6:18

Why we can not customize encode/decode buffer size when initializing a new reader or writer through a function argument.

NSQ use snappy for its message encode/decode with one connection one snappy encoder/decoder. This will consume more and more memory according to the connections number as each connection will use 64KB memory.

Also, this is more friendly for uses cases where the uncompressed data are almost all in small size.

Hi,

this is a follow-up to #30. The same build but for golang 1.6 fails for another reason:

# github.com/golang/snappy
asm: invalid instruction: 00228 (/home/travis/gopath/src/github.com/golang/snappy/encode_amd64.s:338)   MOVWQZX table+120(SP)(R11*2), R15
asm: invalid instruction: 00234 (/home/travis/gopath/src/github.com/golang/snappy/encode_amd64.s:343)   MOVW    AX, table+120(SP)(R11*2)
asm: invalid instruction: 00589 (/home/travis/gopath/src/github.com/golang/snappy/encode_amd64.s:506)   MOVW    AX, table+120(SP)(R11*2)
asm: invalid instruction: 00606 (/home/travis/gopath/src/github.com/golang/snappy/encode_amd64.s:515)   MOVWQZX table+120(SP)(R11*2), R15
asm: invalid instruction: 00610 (/home/travis/gopath/src/github.com/golang/snappy/encode_amd64.s:519)   MOVW    AX, table+120(SP)(R11*2)
asm: asm: assembly of ../../golang/snappy/encode_amd64.s failed

Full log is located here, with the error message starting on line 348 - https://gist.github.com/serejja/b0c40abc844ab4ce1c8630afa988e3bb

Just to provide a bit more info (which is in that log anyway):

1. go version go1.4 linux/amd64
2. Ubuntu 12.04.5 LTS, 3.13.0-29-generic
3. gcc (Ubuntu/Linaro 4.6.3-1ubuntu5) 4.6.3

Please let me know if this is not related to golang/snappy itself and I should open a ticket on Travis CI.
Thanks!

I've encountered an issue with the golang/snappy library where I'm unable to decompress a Snappy compressed JSON file. The error I receive is Failed to decompress content: snappy: corrupt input. However, I've verified that the file is not corrupt by successfully decompressing it using the snzip tool.

Steps to Reproduce:

1. Compress a JSON file using Spark job by using this parameter .option("compression", "snappy") and write it to s3.
2. Attempt to decompress the file from s3 using the following Go code:

package main

import (
	"bytes"
	"fmt"
	"io/ioutil"
	"log"
	"github.com/golang/snappy"
)

func main() {
	// Read the compressed file
	content, err := ioutil.ReadFile("path_to_your_snappy_file.snappy")
	if err != nil {
		log.Fatalf("Failed to read file: %v", err)
	}

	// Decompress using golang/snappy
	decompressed, err := snappy.Decode(nil, content)
	if err != nil {
		log.Fatalf("Failed to decompress content: %v", err)
	}

	// Print the decompressed content
	fmt.Println(string(decompressed))
}

Observe the error: Failed to decompress content: snappy: corrupt input.

Expected Behavior:

The Snappy compressed JSON file should be decompressed without errors.

Actual Behavior:

Received an error indicating the input is corrupt, even though other tools like snzip can decompress the file without issues.

Additional Information:

The Snappy compressed file is a JSON file where each line is a separate JSON object.
I've verified the integrity of the file by decompressing it using snzip.
The issue might be related to the specific Snappy format or framing used, but I'm not certain.

What steps will reproduce the problem?

Call snappy.Decode(nil, x) where x has the form 0xff 0xff 0xff 0xff ... 0xff 
0x00. The function call issues a runtime panic.

What is the expected output? What do you see instead?

No panic and a non-nil error returned, "snappy: corrupt input" or something 
reasonable. This is the behavior if the input has fewer than five 0xff bytes.

What version of the product are you using? On what operating system?

Revision 12e4b4183793 (go1.3.1-linux-amd64)

Please provide any additional information below.

There seem to be panics in three different locations, depending on the length 
of the input. I put together a gist showing the lengths at which each different 
panic occurs.

https://gist.github.com/bmatsuo/2cad6e15c1e3a04d9879

I'm not actually positive these inputs are invalid by the spec. I just stumbled 
into them testing behavior of "github.com/mreiferson/go-snappystream" when 
decoding junk input.

Original issue reported on code.google.com by bryan.matsuo on 8 Sep 2014 at 2:11

version 0.0.2, 0.0.3: crash
version 0.0.1: OK
crashes in encode_arm64.s

error cause is
related to snappy, arm64
runtime.sigpanic()
        /usr/local/go/src/runtime/signal_unix.go:741 +0x230 fp=0x1400159e140 sp=0x1400159e100 pc=0x10303b9f0
github.com/golang/snappy.encodeBlock(0x14000dc7502, 0x1304, 0x1304, 0x14000dc6000, 0x102f, 0x13bc, 0x14000195b01)
        $HOME/go/pkg/mod/github.com/golang/[email protected]/encode_arm64.s:666 +0x360 fp=0x140015a61e0 sp=0x1400159e150 pc=0x1037118c0
github.com/golang/snappy.Encode(0x14000dc7500, 0x1306, 0x1306, 0x0, 0x0, 0x0, 0x2, 0x4, 0x140015a62f8)
       $HOME/go/pkg/mod/github.com/golang/[email protected]/encode.go:39 +0x17c fp=0x140015a6230 sp=0x140015a61e0 pc=0x103710dfc

I'm trying to get a full working repro, but my current best guess is this was while trying to read a file which had experienced a partial write of the compressed data:

panic: runtime error: slice bounds out of range

goroutine 69 [running]:
panic(0xf4cf00, 0xc82000e0b0)
    /go/src/runtime/panic.go:464 +0x3e6
github.com/golang/snappy.(*Reader).Read(0xc89eb43730, 0xcb1ae08000, 0x10000, 0x10000, 0xa, 0x0, 0x0)
    /gopath/src/github.com/golang/snappy/decode.go:198 +0xba4
bufio.(*Reader).fill(0xc9eb4603c0)
    /go/src/bufio/bufio.go:97 +0x1e9
bufio.(*Reader).Read(0xc9eb4603c0, 0xc93f20326d, 0xd93, 0xd93, 0xc82b48fd00, 0x0, 0x0)
    /go/src/bufio/bufio.go:207 +0x260
io/ioutil.(*nopCloser).Read(0xc91fe04ec0, 0xc93f20326d, 0xd93, 0xd93, 0x20e1843c1a693, 0x0, 0x0)
    <autogenerated>:4 +0x82
bufio.(*Scanner).Scan(0xc837df2500, 0x11b6370)
    /go/src/bufio/scan.go:208 +0x97b
...

Hello,

From my simple benchmark for a util that is using snappy/gzip for compression,
I noticed that snappy is 10x faster than gzip in compression, but it's slower by 2x in decompression.
I'm wondering if I'm using incorrectly or whether this is an expected performance.

Thanks,

I looked at the history of snappytool and it seems like the tool was purposely rewritten to use block format instead of stream format. What's the reason behind this? Why can't the snappytool support both stream and block format?

I would love to have stream format support because, then, I can use the tool to decode a snappy encoded stream (that is stored in some DB) for troubleshooting and debugging purposes.

Interested to find out why there's no streaming interface for snappy?  Wouldn't 
the protocol support it?  Would be nice to know that data streams could use 
snappy too like the json stuff: `json.NewEncoder(w).Encode(obj)` 

Original issue reported on code.google.com by [email protected] on 25 Jul 2014 at 5:20

The following program crashes with a panic:

package main

import "github.com/golang/snappy/snappy"

func main() {
    data := []byte("\xff\xff\xff\xff\xff\xff\xff\xff\xec0")
    n, err := snappy.DecodedLen(data)
    if err != nil || n > 1e6 {
        return
    }
    dec, err := snappy.Decode(nil, data)
    if err != nil {
        if dec != nil {
            panic("dec is not nil")
        }
        return
    }
    if len(dec) != n {
        println(len(dec), n)
        panic("bad decoded len")
    }
    n = snappy.MaxEncodedLen(len(dec))
    enc, err := snappy.Encode(nil, dec)
    if err != nil {
        panic(err)
    }
    if len(enc) > n {
        panic("bad encoded len")
    }
}

panic: runtime error: index out of range

goroutine 1 [running]:
github.com/golang/snappy/snappy.Decode(0x0, 0x0, 0x0, 0xc208041f00, 0xa, 0x20, 0x0, 0x0, 0x0, 0x0, ...)
    src/github.com/golang/snappy/snappy/decode.go:54 +0xaca
main.main()
    snappy.go:11 +0xf6

on commit 156a073

As of commit 6880122, I see the following errors. checking out the previous commit resolves.

$ git checkout HEAD^
Previous HEAD position was 45da9d9... Zero out only that part of the hash table in use.
HEAD is now at 6880122... Write the encoder's encodeBlock in asm.
[lloyd@scylla snappy]$ go test
# github.com/golang/snappy
asm: invalid instruction: 00231 (/home/lloyd/dev/go/src/github.com/golang/snappy/encode_amd64.s:335)    MOVWQZX table+120(SP)(R11*2), R15
asm: invalid instruction: 00237 (/home/lloyd/dev/go/src/github.com/golang/snappy/encode_amd64.s:340)    MOVW    AX, table+120(SP)(R11*2)
asm: invalid instruction: 00586 (/home/lloyd/dev/go/src/github.com/golang/snappy/encode_amd64.s:476)    MOVW    AX, table+120(SP)(R11*2)
asm: invalid instruction: 00603 (/home/lloyd/dev/go/src/github.com/golang/snappy/encode_amd64.s:485)    MOVWQZX table+120(SP)(R11*2), R15
asm: invalid instruction: 00607 (/home/lloyd/dev/go/src/github.com/golang/snappy/encode_amd64.s:489)    MOVW    AX, table+120(SP)(R11*2)
asm: asm: assembly of ./encode_amd64.s failed
FAIL    github.com/golang/snappy [build failed]
[lloyd@scylla snappy]$ git checkout HEAD^
Previous HEAD position was 6880122... Write the encoder's encodeBlock in asm.
HEAD is now at 17e4358... Restrict the scope of the tableSize variable.
[lloyd@scylla snappy]$ go test
PASS
ok      github.com/golang/snappy    0.532s

Any chance you'd be willing to tag releases following semver?

I guess adding 5.8 Mb snappy.test in 6ffc20e was a mistake?

If Decode() is called with a nil destination buffer and the uncompressed length 
is zero, then a nil slice will be returned. This contradicts the docs, which 
say that "a newly allocated slice will be returned" in this case.

Repro: http://play.golang.org/p/NiNH49J4M2 . This doesn't actually run because 
the playground can't import snappy, but you can copy/paste this and run it on 
your machine.

There are two alternatives for fixing this problem:

1. If the current behavior is "as designed", update the docs to say that a nil 
slice will be returned of the decompressed output has length 0 and the dst 
input is nil.
2. Add a nil check and allocate a zero-length byte slice to handle this case.

I've attached patches for each of these alternatives (but don't apply them both 
since the two solutions are mutually exclusive).

Original issue reported on code.google.com by [email protected] on 28 Aug 2014 at 5:25

Attachments:

• alloc_zero_length_return.diff
• doc_as_designed.diff

It is great that this snappy golang library is getting some love, especially assembly language love. Nice!

A while back I wrote a Go implementation of the framing/streaming spec for snappy. It is here:
https://github.com/glycerine/go-unsnap-stream

If the maintainers of golang/snappy would like to adopt my implementation and integrate the framing implementation with golang/snappy library or a future Go standard lib for snappy, you have my blessing. (Especially if you want to re-write parts in assembly to make it even faster.)

Thanks again for improving golang/snappy.

I have an application which writes large volume of data to Victoria Metrics using prometheus remote write API. The API specs requires compressing the data being written using snappy encoding. When I do memory profile of my application, the snappy encoding alone is using ~900MB of memory.

This is problematic because I process data in real time using multiple replicas of my application and then write the processed data to Victoria Metrics. So the total memory usage by all replicas becomes too high.

A similar issue, where snappy decode is consuming high memory in prometheus: prometheus/prometheus#13174

With version 0.0.2 on Golang 1.15, the tests are failing on ARMv7hl and i686:

Testing    in: /builddir/build/BUILD/snappy-0.0.2/_build/src
         PATH: /builddir/build/BUILD/snappy-0.0.2/_build/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/sbin
       GOPATH: /builddir/build/BUILD/snappy-0.0.2/_build:/usr/share/gocode
  GO111MODULE: off
      command: go test -buildmode pie -compiler gc -ldflags " -X github.com/golang/snappy/version=0.0.2 -extldflags '-Wl,-z,relro -Wl,--as-needed  -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld  '"
      testing: github.com/golang/snappy
github.com/golang/snappy
--- FAIL: TestDecode (0.00s)
    snappy_test.go:365: #29 (decodedLen=0; tagCopy4, 4 extra length|offset bytes; with msb set (0x93); discovered by go-fuzz):
        got  "", snappy: unsupported literal length
        want "", snappy: corrupt input
FAIL
exit status 1
FAIL	github.com/golang/snappy	0.628s

The comment (and docs) to NewBufferedWriter state that "Users must call Close to guarantee all data has been forwarded to the underlying io.Writer":

However, in looking at the Flush/Close implementations, and their usages in tests, it seems that users only need to call Flush, not Close, which would allow a caller to continue using the buffered writer.

I believe the contributing instructions in the README are no longer accurate. 
They suggest that users follow Go's practice for contributing to snappy-go, 
citing the following link:

  http://golang.org/doc/contribute.html

But the contributing instructions for Go have changed since the switch to git. 
And, it's no longer clear to me how one would submit a patch to this repo for 
code review. Further, while Go now uses the Gerrit system for code review it 
looks like the older Rietveld system may still be in use here? But I just don't 
know for sure.

Original issue reported on code.google.com by bryan.matsuo on 23 Feb 2015 at 7:29

refer: https://code.google.com/p/hadoop-snappy/

The existing Go decoded was no doubt tested with go-fuzz. Does the new asm decoder also need heavy testing with bad inputs?

I saw all example used this code

	rawContent, err := ioutil.ReadFile(localSnappyFilePath)
	if err != nil {
		return err
	}

	content, err := snappy.Decode(nil, rawContent)
	if err != nil {
		return err
	}

but my VM just 1 GB RAM, and the files size is 2GB, i can't read it all into memory

thanks your hard work and your patience 🙏

I tried compressing a dd generated file of bigger size using snappy.
And i found the resulting compressed size to be larger than the actual size.
How to avoid it ?
Is fixed size compression possible?

s/goinstall/go get/

Original issue reported on code.google.com by [email protected] on 17 Jan 2013 at 2:48

code:

    stringReader := strings.NewReader(pbMsg.GetBody().GetReqData().GetContent())
    snappyReader := snappy.NewReader(stringReader)
    rNum, _ := snappyReader.Read(conBytes)

line:pbMsg.GetBody().GetReqData().GetContent() return a string . when using the stringReader as the paramater , the rNum is 0, why? 3q very much

I would find the Encode and Decode functions easier to use if they would extend their dst argument to its capacity. This would let the user optimally reuse a dst buffer when processing multiple blocks, without worrying about reallocating or preallocating.

var dst []byte
src := readSomething()
for len(src) > 0 {
    dst = snappy.Encode(dst, src)
    writeSomething(dst)
    src = readSomething()
}

The changes would be pretty simple, swapping a couple len for cap calls in the functions. However strictly speaking this is a breaking change; if someone is not expecting the codec to touch anything past len(dst) they would be unpleasantly surprised. So I wanted to check if there's interest before I prepare a PR (of course also with documentation updates and a benchmark demoing/justifying the change).

I would like to see some functions to facilitate building a framed snappy 
stream. Some functions that would be helpful:

EncodeFrame(dst, src []byte)
StartFrames(dst)
FinishFrames(dst)

Please provide any additional information below.

The official snappy repository has a document describing the framing format for 
snappy (http://code.google.com/p/snappy/source/browse/trunk/framing_format.txt) 

Original issue reported on code.google.com by jehiah on 19 Jul 2012 at 3:58

Hi team, we met an error snappy: corrupt input while using snappy to compress through a TCP connection.

How we build the connection:

On the source side:

conn.reader = snappy.NewReader(io.Reader(net.Conn))
conn.writeCloser = snappy.NewBufferedWriter(io.WriteCloser(net.Conn))
conn.Write(buf)

On the destination side:

conn.reader = snappy.NewReader(io.Reader(net.Conn))
conn.writeCloser = snappy.NewBufferedWriter(io.WriteCloser(net.Conn))
conn.Read(buf)

We get the error snappy: corrupt input when conn.Read(buf). And this error happens intermittently.

Is this caused by network problem?

We read some snappy code, and we know that this error is reported because the checksum or the decode result length is wrong. But we use snappy based on a TCP connection and TCP can guarantee data integrity. So if snappy needs several network packages to decode the complete data? Or this is caused by some problem with the network card, the hardware cannot verify the data correctly.

Is this caused by memory problem?

I also find some discussions on the network that suspect this is caused by memory overflow or runtime memory limit not enough. But I cannot make sure. Since we don't get any other error messages.

I'd appreciate any help or suggestions on how to debug. Thanks!

golang

java

Can you please include the file needed for testing (snapp_test.go):

        words, err := ioutil.ReadFile("/usr/share/dict/words")

Original issue reported on code.google.com by [email protected] on 14 Nov 2011 at 8:05

Hey,

I try to use snappy with golang, I see JS and Java has a function named isValidCompressed which can validate the encoded buffer, but I can't see that function with golang.
I am thinking to use decode function directly. And if it returns error, it means the encoded buffer is invalid. But I am not sure that's safe/correct or not.
Such as below:
byte[] bytes = Snappy.isValidCompressedBuffer(recordBytes) ? Snappy.uncompress(recordBytes) : recordBytes;
Anyone have any ideas on how to safe and quickly validate the encoded buffer?

Hi,
why is the encoded (compressed) string larger than the original?
https://play.golang.org/p/J7iL2AwQ37

go get github.com/golang/snappy

src/github.com/golang/snappy/decode_amd64.s: Assembler messages:
src/github.com/golang/snappy/decode_amd64.s:34: Error: no such instruction: text ·decode(SB),NOSPLIT,$48-56' src/github.com/golang/snappy/decode_amd64.s:36: Error: junk(FP)' after expression
src/github.com/golang/snappy/decode_amd64.s:36: Error: too many memory references for movq' src/github.com/golang/snappy/decode_amd64.s:37: Error: junk(FP)' after expression
src/github.com/golang/snappy/decode_amd64.s:37: Error: too many memory references for movq' src/github.com/golang/snappy/decode_amd64.s:38: Error: too many memory references formovq'
src/github.com/golang/snappy/decode_amd64.s:39: Error: too many memory references for movq' src/github.com/golang/snappy/decode_amd64.s:40: Error: too many memory references foradd'
src/github.com/golang/snappy/decode_amd64.s:41: Error: junk `(FP)' after expression

go version

go version xgcc (Ubuntu 4.9.1-0ubuntu1) 4.9.1 linux/amd64

Getting this error upon attempting to run:

`go get github.com/golang/snappy

github.com/golang/snappy

asm: 00001 (/root/go/src/github.com/golang/snappy/encode_arm64.s:30) TEXT "".emitLiteral(SB), NOSPLIT, $32-56: unaligned frame size 32 - must be 8 mod 16 (or 0)
asm: 00119 (/root/go/src/github.com/golang/snappy/encode_arm64.s:264) TEXT "".encodeBlock(SB), $32896-56: unaligned frame size 32896 - must be 8 mod 16 (or 0)
asm: assembly failed`

Hello,
When I try to test the cmd/influxd/run in the project github.com/influxdata/influxdb in version 1.4.2 it fails with

github.com/golang/snappy

/usr/share/gocode/src/github.com/golang/snappy/decode_amd64.go:14: missing function body for "decode"
/usr/share/gocode/src/github.com/golang/snappy/encode_amd64.go:14: missing function body for "emitLiteral"
/usr/share/gocode/src/github.com/golang/snappy/encode_amd64.go:19: missing function body for "emitCopy"
/usr/share/gocode/src/github.com/golang/snappy/encode_amd64.go:24: missing function body for "extendMatch"
/usr/share/gocode/src/github.com/golang/snappy/encode_amd64.go:29: missing function body for "encodeBlock"

Thank you for your help

Will this project be migrating to github as code.google.com will be shutting 
down in January 2016?

Original issue reported on code.google.com by jehiah on 16 Apr 2015 at 1:53