kothar / brotli-go Goto Github PK

View Code? Open in Web Editor NEW

82.0 82.0 9.0 7.4 MB

Not maintained -> see itch.io's more up-to-date fork 👉

Home Page: https://github.com/itchio/go-brotli

License: MIT License

Go 1.27% Makefile 0.02% C 7.36% C++ 91.35%

brotli-go's People

Contributors

Stargazers

Watchers

Forkers

laysakura itchio nelson69 go-siris millisecond jonike golint-fixer mgbolivierdahinden

brotli-go's Issues

Reader doesn't seem to be flushing properly

Nice work on this cgo wrapper :) I'm using it as a comparison to benchmark my implementation.

In the meantime, the Reader is crashing my unit tests. (I verified that the writer is working since my decoder can read the output).

--- FAIL: TestCGoRoundTripBrotli (0.26s)
    common_test.go:59: test 0, zeros.bin:6:1000000: mismatching count: got 32768, want 1000000
    common_test.go:62: test 0, zeros.bin:6:1000000: mismatching checksum: got 0x011ffca6, want 0x0f410f3f
    common_test.go:59: test 1, random.bin:6:1000000: mismatching count: got 262144, want 1000000
    common_test.go:62: test 1, random.bin:6:1000000: mismatching checksum: got 0x91beb9df, want 0xcd5e034f
    common_test.go:59: test 2, binary.bin:6:1000000: mismatching count: got 65536, want 1000000
    common_test.go:62: test 2, binary.bin:6:1000000: mismatching checksum: got 0x681cc440, want 0xfc1091d8
    common_test.go:59: test 3, huffman.txt:6:1000000: mismatching count: got 229376, want 1000000
    common_test.go:62: test 3, huffman.txt:6:1000000: mismatching checksum: got 0x2cdba6dd, want 0x44a993e0
    common_test.go:59: test 4, digits.txt:6:1000000: mismatching count: got 131072, want 1000000
    common_test.go:62: test 4, digits.txt:6:1000000: mismatching checksum: got 0xd2a117ed, want 0x41db1af3
    common_test.go:59: test 5, twain.txt:6:1000000: mismatching count: got 98304, want 1000000
    common_test.go:62: test 5, twain.txt:6:1000000: mismatching checksum: got 0x3f8120eb, want 0xe6a0db63
FAIL
exit status 1
FAIL    _/home/rawr/Projects/compress/internal/benchmark    1.239s

The fact that its only outputting data the size of 1<<15 indicates to me that its some type of flushing issue.

How to reproduce:

go get github.com/dsnet/compress/internal/benchmark
go test github.com/dsnet/compress/internal/benchmark

Continuous integration

Just opening this so I don't forget about it when #4 is over

Flush support

When using brotli as transport compression, it's useful to be able to force flush between messages.

edit: compression, not encryption 🐙

Travis CI build using mixed of 'local clone' and 'whatever is on master' b/c of gopkg.in

If you look carefully at travis' log output (comments mine);

# make canonical import directory (or so travis's go setup thinks)
$ mkdir -p $HOME/gopath/src/github.com/kothar/brotli-go

# copy everything from original build dir to there
$ rsync -az ${TRAVIS_BUILD_DIR}/ $HOME/gopath/src/github.com/kothar/brotli-go/

# set `TRAVIS_BUILD_DIR` environment variable to our canonical import path
$ export TRAVIS_BUILD_DIR=$HOME/gopath/src/github.com/kothar/brotli-go

# cd there
$ cd $HOME/gopath/src/github.com/kothar/brotli-go

It thinks github.com/kothar/brotli-go is the canonical import path, whereas gopkg.in/kothar/brotli-go.v0 is.

This means, for example, that when running a travis build of a branch (for a pending pull request), the test files are those of the branch, but the enc, dec, shared modules are those of whatever is on master (because it fetches gopkg.in/kothar/brotli-go.v0 separately)

I'll submit a PR to fix this, but we'll have to be careful once v1 comes around (I can think of something based on $TRAVIS_TAG to support v1+ but it won't be pretty)

Custom dictionaries

I created a fork with custom dictionary support (https://github.com/crawlcoin/brotli-go) and would like to merge it back upstream, open to a pull request with this support?

assertion failed in WriteBits

Hey, not sure if this is an issue in google/brotli, this, or the way I've used it but I was wondering if you could give a look at itchio/butler#3

You can browse the v0.4.5 tree at https://github.com/itchio/butler/tree/v0.4.5 — the testBrotli function lives at https://github.com/itchio/butler/blob/v0.4.5/butler.go#L151

Basically you should just be able to feed butler.exe (the v0.4.5 windows-amd64 build, which you can grab from this .7z archive) to a compress/decompress test & be able to reproduce.

I'll try to come up with a stack trace & a better/more self-contained test case later on, but if you've already encountered that / have any idea where it might come from, I'd be glad to know about it. Thanks!

Streaming interface

I'm currently looking into supporting BrotliDecompressStream and its compress equivalent, and I had a simple question: the README mentions the need to wrap some C++ structures, but (as of 10 days ago), BrotliDecompressStream & friends seem to be usable purely from the C interface?

It looks relatively easy to do but I'm a cgo noob so it might take me a day or two.

(unsafe.)Pointers welcome!

P.S: do tell me if this issue is unwelcome noise on the issue tracker, in which case I'll try & get it working in my corner and come back with a PR later — just thought I'd deduplicate work in case someone is already on it!

Roundtrip stream test fails with random 2MB input

See itchio@584d625 (branch stream-fail of https://github.com/itchio/brotli-go)

I probably messed something when coding the streaming interface, will keep investigating, but if @kothar / @dsnet / anyone wants to take a closer look, be my guest!

index out of range when availableIn=0 and r.needOutput is true

Trace:

r.bufferRead = 131072, r.availableIn = 0, inputPosition = 131072, buffer len = 131072
panic: runtime error: index out of range

goroutine 140 [running]:
github.com/itchio/butler/vendor/gopkg.in/kothar/brotli-go.v0/dec.(*BrotliReader).Read(0xc42016e150, 0xc42011d640, 0x1, 0x1, 0x1, 0x0, 0x0)
        /home/amos/go/src/github.com/itchio/butler/vendor/gopkg.in/kothar/brotli-go.v0/dec/decode.go:143 +0x62d
github.com/itchio/butler/vendor/github.com/itchio/wharf/wire.(*ReadContext).ReadByte(0xc42001b980, 0x10080, 0x0, 0x0)
        /home/amos/go/src/github.com/itchio/butler/vendor/github.com/itchio/wharf/wire/read.go:33 +0x52
encoding/binary.ReadUvarint(0xf915c0, 0xc42001b980, 0x4138ed, 0x9ac080, 0xc420010300)
        /usr/local/go/src/encoding/binary/varint.go:110 +0x4a
github.com/itchio/butler/vendor/github.com/itchio/wharf/wire.(*ReadContext).ReadMessage(0xc42001b980, 0xf97bc0, 0xc4201961c0, 0x0, 0x0)
        /home/amos/go/src/github.com/itchio/butler/vendor/github.com/itchio/wharf/wire/read.go:63 +0x3b
github.com/itchio/butler/vendor/github.com/itchio/wharf/pwr.readOps(0xc42001b980, 0xc420010300, 0xc4200102a0)
        /home/amos/go/src/github.com/itchio/butler/vendor/github.com/itchio/wharf/pwr/apply.go:936 +0x114
created by github.com/itchio/butler/vendor/github.com/itchio/wharf/pwr.(*ApplyContext).patchAll
        /home/amos/go/src/github.com/itchio/butler/vendor/github.com/itchio/wharf/pwr/apply.go:466 +0x6ff
amos@poppy:~/butler-tests/possessed$

Offending bit of code:

                if r.availableIn > 0 || r.needOutput {
                        // Decompress
                        inputPosition := r.bufferRead - int(r.availableIn)
                        fmt.Printf("r.bufferRead = %d, r.availableIn = %d, inputPosition = %d, buffer len = %d\n", r.bufferRead, r.availableIn, inputPosition, len(r.buffer));
                        result := C.BrotliDecompressStream_Wrapper(
                                &r.availableIn,
                                (*C.uint8_t)(unsafe.Pointer(&r.buffer[inputPosition])),
                                &availableOut,
                                (*C.uint8_t)(unsafe.Pointer(&p[0])),
                                &r.totalOut,
                                (*C.BrotliState)(r.state),
                        )

                        n = maxOutput - int(availableOut)

I think the issue is that.. &r.buffer[inputPosition] is valid in C because there's no bound check going on, but in golang there is, and it fails. I wonder if unsafe.Pointer(&r.buffer)+inputPosition would work (and be faster!

If that fails, maybe we can pass a null pointer instead in that very case.

I'm impressed this even happened though (I have a reproducible test case here, but it involves a >2GB file right now), I've been using brotli-go in production for 1yr+ without issues :)

uptdate to last google/brotli?

Can you please update to last google/brotli?

EOF behavior

When reading the entire contents in a single Read call, BrotliReader returns io.EOF.

This differs from a bytes.Reader for example, see this self-contained test:

package pwr_test

import (
    "bytes"
    "io"
    "testing"

    "gopkg.in/kothar/brotli-go.v0/dec"
    "gopkg.in/kothar/brotli-go.v0/enc"
)

func must(t *testing.T, err error) {
    if err != nil {
        t.Fatal(err.Error())
        t.FailNow()
    }
}

func makeBrotliReader(t *testing.T, data []byte) io.Reader {
    params := enc.NewBrotliParams()
    params.SetQuality(1)

    buf := new(bytes.Buffer)
    writer := enc.NewBrotliWriter(params, buf)
    _, err := writer.Write(data)
    must(t, err)

    err = writer.Close()
    must(t, err)

    return dec.NewBrotliReader(bytes.NewReader(buf.Bytes()))
}

func Test_BrotliEOFBehavior(t *testing.T) {
    buf := []byte{0x1, 0x2, 0x3, 0x4}

    normalReader := bytes.NewReader(buf)
    brotliReader := makeBrotliReader(t, buf)

    receiver := make([]byte, len(buf))

    _, err := normalReader.Read(receiver)
    t.Log("Normal read:")
    must(t, err)

    _, err = brotliReader.Read(receiver)
    t.Log("Brotli read:")
    must(t, err)
}

The results are:

$ go test -v ./pwr/brotli_eof_test.go
=== RUN   Test_BrotliEOFBehavior
--- FAIL: Test_BrotliEOFBehavior (0.07s)
        brotli_eof_test.go:43: Normal read:
        brotli_eof_test.go:47: Brotli read:
        brotli_eof_test.go:14: EOF
FAIL
exit status 1

Go GC is not happy

Using go1.5.1

runtime:objectstart Span weird: p=0xc8206a6000 k=0x6410353 s.start=0xc8206a6000 s.limit=0xc8205c0000 s.state=2
fatal error: objectstart: bad pointer in unexpected span

goroutine 8 [running]:
runtime.throw(0x6c9b20, 0x2b)
    /usr/local/go/src/runtime/panic.go:527 +0x90 fp=0xc820030e18 sp=0xc820030e00
runtime.heapBitsForObject(0xc8206a6000, 0x0, 0x0, 0xc800000000, 0x7f1e1595fe30)
    /usr/local/go/src/runtime/mbitmap.go:217 +0x287 fp=0xc820030e50 sp=0xc820030e18
runtime.scanobject(0xc8205ae000, 0xc820025c20)
    /usr/local/go/src/runtime/mgcmark.go:878 +0x239 fp=0xc820030f20 sp=0xc820030e50
runtime.gcDrain(0xc820025c20, 0x7d0)
    /usr/local/go/src/runtime/mgcmark.go:689 +0xf4 fp=0xc820030f50 sp=0xc820030f20
runtime.gcBgMarkWorker(0xc820024a00)
    /usr/local/go/src/runtime/mgc.go:1320 +0x281 fp=0xc820030fb8 sp=0xc820030f50
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc820030fc0 sp=0xc820030fb8
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1239 +0x93

goroutine 1 [chan receive]:
testing.RunTests(0x6de8a8, 0x97daa0, 0x3, 0x3, 0xc820053e01)
    /usr/local/go/src/testing/testing.go:562 +0x8ad
testing.(*M).Run(0xc820045f08, 0xc820073520)
    /usr/local/go/src/testing/testing.go:494 +0x70
main.main()
    _/home/rawr/Projects/compress/internal/benchmark/_test/_testmain.go:58 +0x116

goroutine 17 [syscall, locked to thread]:
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1696 +0x1

goroutine 5 [runnable, locked to thread]:
gopkg.in/kothar/brotli-go.v0/dec._Cfunc_BrotliDecompressStream(0xc8205afb10, 0xc8205afb18, 0xc8205a4000, 0xc8205a6000, 0xc8205afb20, 0xc8205ae000, 0x2)
    ??:0 +0x39
gopkg.in/kothar/brotli-go.v0/dec.(*BrotliReader).Read(0xc8205ae000, 0xc82022c000, 0x8000, 0x8000, 0x8000, 0x0, 0x0)
    /home/rawr/.go/src/gopkg.in/kothar/brotli-go.v0/dec/decode.go:122 +0x38e
io.copyBuffer(0x7f1e13d379b0, 0xc82057e100, 0x7f1e13d379d8, 0xc8205ae000, 0xc82022c000, 0x8000, 0x8000, 0x10000, 0x0, 0x0)
    /usr/local/go/src/io/io.go:381 +0x247
io.Copy(0x7f1e13d379b0, 0xc82057e100, 0x7f1e13d379d8, 0xc8205ae000, 0xc8205ae000, 0x0, 0x0)
    /usr/local/go/src/io/io.go:351 +0x64
_/home/rawr/Projects/compress/internal/benchmark.testRoundTrip(0xc82009c120, 0x6de6e8, 0x6de6f0)
    /home/rawr/Projects/compress/internal/benchmark/common_test.go:47 +0x102e
_/home/rawr/Projects/compress/internal/benchmark.TestCGoRoundTripBrotli(0xc82009c120)
    /home/rawr/Projects/compress/internal/benchmark/cgo_brotlib_test.go:12 +0x117
testing.tRunner(0xc82009c120, 0x97daa0)
    /usr/local/go/src/testing/testing.go:456 +0x98
created by testing.RunTests
    /usr/local/go/src/testing/testing.go:561 +0x86d
exit status 2
FAIL    _/home/rawr/Projects/compress/internal/benchmark    0.094s

I actually struggled with a very similar problem when I was wrapping zlib.
You might want to take a look at what cgzip did to make their cgo stuff happy with go1.5. Specifically the comment here.

It won't cross compile for Linux/amd64 on MacOs.

Error message on MacOS:
../../../gopkg.in/kothar/brotli-go.v0/dec/decode.go:36:2: no buildable Go source files in /Users/mehmetgerceker/workspace/goWorkspace/src/gopkg.in/kothar/brotli-go.v0/shared

Error message on Linux:
main.go:14:2: C source files not allowed when not using cgo or SWIG: bit_reader.c decode.c huffman.c state.c streams.c
../../../gopkg.in/kothar/brotli-go.v0/dec/decode.go:36:2: no buildable Go source files in /home/mehmetg/gohome/src/gopkg.in/kothar/brotli-go.v0/shared

Project: github.com/mehmetg/simpleJSONService

cannot use p.c (type C.struct_CBrotliParams) as type C.CBrotliParams in function argument

i get this error when compiling on ubuntu 14.04.1 ; using go get github.com/kothar/brotli-go
here is the error;
go/src/gopkg.in/kothar/brotli-go.v0/enc/encode.go:148: cannot use p.c (type C.struct_CBrotliParams) as type C.CBrotliParams in function argument go/src/gopkg.in/kothar/brotli-go.v0/enc/encode.go:170: cannot use params.c (type C.struct_CBrotliParams) as type C.CBrotliParams in function argument go/src/gopkg.in/kothar/brotli-go.v0/enc/encode.go:188: cannot use params.c (type C.struct_CBrotliParams) as type C.CBrotliParams in function argument
i have installed build-essentials and gcc. i also get this error on any other go project that uses brotli, like
https://github.com/barnacs/compy

cgo argument has Go pointer to Go pointer (go 1.6)

With

$ go version
go version go1.6rc2 windows/amd64

...running tests fail:

=== RUN   TestStreamDecompression
--- FAIL: TestStreamDecompression (0.05s)
panic: runtime error: cgo argument has Go pointer to Go pointer [recovered]
        panic: runtime error: cgo argument has Go pointer to Go pointer

goroutine 5 [running]:
testing.tRunner.func1(0xc082076000)
        C:/Go/src/testing/testing.go:467 +0x199
gopkg.in/kothar/brotli-go.v0/dec.NewBrotliReaderSize(0xb8a6b0, 0xc082007f80, 0x20000, 0xb8a6b0)
        C:/msys64/home/amwenger/Dev/go/src/gopkg.in/kothar/brotli-go.v0/dec/decode.go:208 +0x130
gopkg.in/kothar/brotli-go.v0/dec.NewBrotliReader(0xb8a6b0, 0xc082007f80, 0x758ff0)
        C:/msys64/home/amwenger/Dev/go/src/gopkg.in/kothar/brotli-go.v0/dec/decode.go:195 +0x3b
gopkg.in/kothar/brotli-go.v0/dec.TestStreamDecompression(0xc082076000)
        C:/msys64/home/amwenger/Dev/go/src/gopkg.in/kothar/brotli-go.v0/dec/decode_test.go:28 +0x3c5
testing.tRunner(0xc082076000, 0x6ba5c0)
        C:/Go/src/testing/testing.go:473 +0x9f
created by testing.RunTests
        C:/Go/src/testing/testing.go:582 +0x899
FAIL    gopkg.in/kothar/brotli-go.v0/dec        0.140s

I'm starting to think that we should wrap the brotli interface into something sane, ie. something that returns number of bytes consumed, etc. rather than moving around a bunch of pointers.