kothar / brotli-go Goto Github PK
View Code? Open in Web Editor NEWNot maintained -> see itch.io's more up-to-date fork ๐
Home Page: https://github.com/itchio/go-brotli
License: MIT License
Not maintained -> see itch.io's more up-to-date fork ๐
Home Page: https://github.com/itchio/go-brotli
License: MIT License
Nice work on this cgo wrapper :) I'm using it as a comparison to benchmark my implementation.
In the meantime, the Reader is crashing my unit tests. (I verified that the writer is working since my decoder can read the output).
--- FAIL: TestCGoRoundTripBrotli (0.26s)
common_test.go:59: test 0, zeros.bin:6:1000000: mismatching count: got 32768, want 1000000
common_test.go:62: test 0, zeros.bin:6:1000000: mismatching checksum: got 0x011ffca6, want 0x0f410f3f
common_test.go:59: test 1, random.bin:6:1000000: mismatching count: got 262144, want 1000000
common_test.go:62: test 1, random.bin:6:1000000: mismatching checksum: got 0x91beb9df, want 0xcd5e034f
common_test.go:59: test 2, binary.bin:6:1000000: mismatching count: got 65536, want 1000000
common_test.go:62: test 2, binary.bin:6:1000000: mismatching checksum: got 0x681cc440, want 0xfc1091d8
common_test.go:59: test 3, huffman.txt:6:1000000: mismatching count: got 229376, want 1000000
common_test.go:62: test 3, huffman.txt:6:1000000: mismatching checksum: got 0x2cdba6dd, want 0x44a993e0
common_test.go:59: test 4, digits.txt:6:1000000: mismatching count: got 131072, want 1000000
common_test.go:62: test 4, digits.txt:6:1000000: mismatching checksum: got 0xd2a117ed, want 0x41db1af3
common_test.go:59: test 5, twain.txt:6:1000000: mismatching count: got 98304, want 1000000
common_test.go:62: test 5, twain.txt:6:1000000: mismatching checksum: got 0x3f8120eb, want 0xe6a0db63
FAIL
exit status 1
FAIL _/home/rawr/Projects/compress/internal/benchmark 1.239s
The fact that its only outputting data the size of 1<<15 indicates to me that its some type of flushing issue.
How to reproduce:
go get github.com/dsnet/compress/internal/benchmark
go test github.com/dsnet/compress/internal/benchmark
Just opening this so I don't forget about it when #4 is over
When using brotli as transport compression, it's useful to be able to force flush between messages.
edit: compression, not encryption ๐
If you look carefully at travis' log output (comments mine);
# make canonical import directory (or so travis's go setup thinks)
$ mkdir -p $HOME/gopath/src/github.com/kothar/brotli-go
# copy everything from original build dir to there
$ rsync -az ${TRAVIS_BUILD_DIR}/ $HOME/gopath/src/github.com/kothar/brotli-go/
# set `TRAVIS_BUILD_DIR` environment variable to our canonical import path
$ export TRAVIS_BUILD_DIR=$HOME/gopath/src/github.com/kothar/brotli-go
# cd there
$ cd $HOME/gopath/src/github.com/kothar/brotli-go
It thinks github.com/kothar/brotli-go
is the canonical import path, whereas gopkg.in/kothar/brotli-go.v0
is.
This means, for example, that when running a travis build of a branch (for a pending pull request), the test files are those of the branch, but the enc
, dec
, shared
modules are those of whatever is on master (because it fetches gopkg.in/kothar/brotli-go.v0
separately)
I'll submit a PR to fix this, but we'll have to be careful once v1 comes around (I can think of something based on $TRAVIS_TAG
to support v1+ but it won't be pretty)
I created a fork with custom dictionary support (https://github.com/crawlcoin/brotli-go) and would like to merge it back upstream, open to a pull request with this support?
Hey, not sure if this is an issue in google/brotli, this, or the way I've used it but I was wondering if you could give a look at itchio/butler#3
You can browse the v0.4.5 tree at https://github.com/itchio/butler/tree/v0.4.5 โ the testBrotli
function lives at https://github.com/itchio/butler/blob/v0.4.5/butler.go#L151
Basically you should just be able to feed butler.exe
(the v0.4.5 windows-amd64 build, which you can grab from this .7z archive) to a compress/decompress test & be able to reproduce.
I'll try to come up with a stack trace & a better/more self-contained test case later on, but if you've already encountered that / have any idea where it might come from, I'd be glad to know about it. Thanks!
I'm currently looking into supporting BrotliDecompressStream
and its compress equivalent, and I had a simple question: the README mentions the need to wrap some C++ structures, but (as of 10 days ago), BrotliDecompressStream
& friends seem to be usable purely from the C interface?
It looks relatively easy to do but I'm a cgo noob so it might take me a day or two.
(unsafe.)Pointers welcome!
P.S: do tell me if this issue is unwelcome noise on the issue tracker, in which case I'll try & get it working in my corner and come back with a PR later โ just thought I'd deduplicate work in case someone is already on it!
See itchio@584d625 (branch stream-fail of https://github.com/itchio/brotli-go)
I probably messed something when coding the streaming interface, will keep investigating, but if @kothar / @dsnet / anyone wants to take a closer look, be my guest!
Trace:
r.bufferRead = 131072, r.availableIn = 0, inputPosition = 131072, buffer len = 131072
panic: runtime error: index out of range
goroutine 140 [running]:
github.com/itchio/butler/vendor/gopkg.in/kothar/brotli-go.v0/dec.(*BrotliReader).Read(0xc42016e150, 0xc42011d640, 0x1, 0x1, 0x1, 0x0, 0x0)
/home/amos/go/src/github.com/itchio/butler/vendor/gopkg.in/kothar/brotli-go.v0/dec/decode.go:143 +0x62d
github.com/itchio/butler/vendor/github.com/itchio/wharf/wire.(*ReadContext).ReadByte(0xc42001b980, 0x10080, 0x0, 0x0)
/home/amos/go/src/github.com/itchio/butler/vendor/github.com/itchio/wharf/wire/read.go:33 +0x52
encoding/binary.ReadUvarint(0xf915c0, 0xc42001b980, 0x4138ed, 0x9ac080, 0xc420010300)
/usr/local/go/src/encoding/binary/varint.go:110 +0x4a
github.com/itchio/butler/vendor/github.com/itchio/wharf/wire.(*ReadContext).ReadMessage(0xc42001b980, 0xf97bc0, 0xc4201961c0, 0x0, 0x0)
/home/amos/go/src/github.com/itchio/butler/vendor/github.com/itchio/wharf/wire/read.go:63 +0x3b
github.com/itchio/butler/vendor/github.com/itchio/wharf/pwr.readOps(0xc42001b980, 0xc420010300, 0xc4200102a0)
/home/amos/go/src/github.com/itchio/butler/vendor/github.com/itchio/wharf/pwr/apply.go:936 +0x114
created by github.com/itchio/butler/vendor/github.com/itchio/wharf/pwr.(*ApplyContext).patchAll
/home/amos/go/src/github.com/itchio/butler/vendor/github.com/itchio/wharf/pwr/apply.go:466 +0x6ff
amos@poppy:~/butler-tests/possessed$
Offending bit of code:
if r.availableIn > 0 || r.needOutput {
// Decompress
inputPosition := r.bufferRead - int(r.availableIn)
fmt.Printf("r.bufferRead = %d, r.availableIn = %d, inputPosition = %d, buffer len = %d\n", r.bufferRead, r.availableIn, inputPosition, len(r.buffer));
result := C.BrotliDecompressStream_Wrapper(
&r.availableIn,
(*C.uint8_t)(unsafe.Pointer(&r.buffer[inputPosition])),
&availableOut,
(*C.uint8_t)(unsafe.Pointer(&p[0])),
&r.totalOut,
(*C.BrotliState)(r.state),
)
n = maxOutput - int(availableOut)
I think the issue is that.. &r.buffer[inputPosition]
is valid in C because there's no bound check going on, but in golang there is, and it fails. I wonder if unsafe.Pointer(&r.buffer)+inputPosition
would work (and be faster!
If that fails, maybe we can pass a null pointer instead in that very case.
I'm impressed this even happened though (I have a reproducible test case here, but it involves a >2GB file right now), I've been using brotli-go in production for 1yr+ without issues :)
Can you please update to last google/brotli?
When reading the entire contents in a single Read
call, BrotliReader returns io.EOF
.
This differs from a bytes.Reader
for example, see this self-contained test:
package pwr_test
import (
"bytes"
"io"
"testing"
"gopkg.in/kothar/brotli-go.v0/dec"
"gopkg.in/kothar/brotli-go.v0/enc"
)
func must(t *testing.T, err error) {
if err != nil {
t.Fatal(err.Error())
t.FailNow()
}
}
func makeBrotliReader(t *testing.T, data []byte) io.Reader {
params := enc.NewBrotliParams()
params.SetQuality(1)
buf := new(bytes.Buffer)
writer := enc.NewBrotliWriter(params, buf)
_, err := writer.Write(data)
must(t, err)
err = writer.Close()
must(t, err)
return dec.NewBrotliReader(bytes.NewReader(buf.Bytes()))
}
func Test_BrotliEOFBehavior(t *testing.T) {
buf := []byte{0x1, 0x2, 0x3, 0x4}
normalReader := bytes.NewReader(buf)
brotliReader := makeBrotliReader(t, buf)
receiver := make([]byte, len(buf))
_, err := normalReader.Read(receiver)
t.Log("Normal read:")
must(t, err)
_, err = brotliReader.Read(receiver)
t.Log("Brotli read:")
must(t, err)
}
The results are:
$ go test -v ./pwr/brotli_eof_test.go
=== RUN Test_BrotliEOFBehavior
--- FAIL: Test_BrotliEOFBehavior (0.07s)
brotli_eof_test.go:43: Normal read:
brotli_eof_test.go:47: Brotli read:
brotli_eof_test.go:14: EOF
FAIL
exit status 1
Using go1.5.1
runtime:objectstart Span weird: p=0xc8206a6000 k=0x6410353 s.start=0xc8206a6000 s.limit=0xc8205c0000 s.state=2
fatal error: objectstart: bad pointer in unexpected span
goroutine 8 [running]:
runtime.throw(0x6c9b20, 0x2b)
/usr/local/go/src/runtime/panic.go:527 +0x90 fp=0xc820030e18 sp=0xc820030e00
runtime.heapBitsForObject(0xc8206a6000, 0x0, 0x0, 0xc800000000, 0x7f1e1595fe30)
/usr/local/go/src/runtime/mbitmap.go:217 +0x287 fp=0xc820030e50 sp=0xc820030e18
runtime.scanobject(0xc8205ae000, 0xc820025c20)
/usr/local/go/src/runtime/mgcmark.go:878 +0x239 fp=0xc820030f20 sp=0xc820030e50
runtime.gcDrain(0xc820025c20, 0x7d0)
/usr/local/go/src/runtime/mgcmark.go:689 +0xf4 fp=0xc820030f50 sp=0xc820030f20
runtime.gcBgMarkWorker(0xc820024a00)
/usr/local/go/src/runtime/mgc.go:1320 +0x281 fp=0xc820030fb8 sp=0xc820030f50
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc820030fc0 sp=0xc820030fb8
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1239 +0x93
goroutine 1 [chan receive]:
testing.RunTests(0x6de8a8, 0x97daa0, 0x3, 0x3, 0xc820053e01)
/usr/local/go/src/testing/testing.go:562 +0x8ad
testing.(*M).Run(0xc820045f08, 0xc820073520)
/usr/local/go/src/testing/testing.go:494 +0x70
main.main()
_/home/rawr/Projects/compress/internal/benchmark/_test/_testmain.go:58 +0x116
goroutine 17 [syscall, locked to thread]:
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1696 +0x1
goroutine 5 [runnable, locked to thread]:
gopkg.in/kothar/brotli-go.v0/dec._Cfunc_BrotliDecompressStream(0xc8205afb10, 0xc8205afb18, 0xc8205a4000, 0xc8205a6000, 0xc8205afb20, 0xc8205ae000, 0x2)
??:0 +0x39
gopkg.in/kothar/brotli-go.v0/dec.(*BrotliReader).Read(0xc8205ae000, 0xc82022c000, 0x8000, 0x8000, 0x8000, 0x0, 0x0)
/home/rawr/.go/src/gopkg.in/kothar/brotli-go.v0/dec/decode.go:122 +0x38e
io.copyBuffer(0x7f1e13d379b0, 0xc82057e100, 0x7f1e13d379d8, 0xc8205ae000, 0xc82022c000, 0x8000, 0x8000, 0x10000, 0x0, 0x0)
/usr/local/go/src/io/io.go:381 +0x247
io.Copy(0x7f1e13d379b0, 0xc82057e100, 0x7f1e13d379d8, 0xc8205ae000, 0xc8205ae000, 0x0, 0x0)
/usr/local/go/src/io/io.go:351 +0x64
_/home/rawr/Projects/compress/internal/benchmark.testRoundTrip(0xc82009c120, 0x6de6e8, 0x6de6f0)
/home/rawr/Projects/compress/internal/benchmark/common_test.go:47 +0x102e
_/home/rawr/Projects/compress/internal/benchmark.TestCGoRoundTripBrotli(0xc82009c120)
/home/rawr/Projects/compress/internal/benchmark/cgo_brotlib_test.go:12 +0x117
testing.tRunner(0xc82009c120, 0x97daa0)
/usr/local/go/src/testing/testing.go:456 +0x98
created by testing.RunTests
/usr/local/go/src/testing/testing.go:561 +0x86d
exit status 2
FAIL _/home/rawr/Projects/compress/internal/benchmark 0.094s
I actually struggled with a very similar problem when I was wrapping zlib.
You might want to take a look at what cgzip did to make their cgo stuff happy with go1.5. Specifically the comment here.
Error message on MacOS:
../../../gopkg.in/kothar/brotli-go.v0/dec/decode.go:36:2: no buildable Go source files in /Users/mehmetgerceker/workspace/goWorkspace/src/gopkg.in/kothar/brotli-go.v0/shared
Error message on Linux:
main.go:14:2: C source files not allowed when not using cgo or SWIG: bit_reader.c decode.c huffman.c state.c streams.c
../../../gopkg.in/kothar/brotli-go.v0/dec/decode.go:36:2: no buildable Go source files in /home/mehmetg/gohome/src/gopkg.in/kothar/brotli-go.v0/shared
Project: github.com/mehmetg/simpleJSONService
i get this error when compiling on ubuntu 14.04.1 ; using go get github.com/kothar/brotli-go
here is the error;
go/src/gopkg.in/kothar/brotli-go.v0/enc/encode.go:148: cannot use p.c (type C.struct_CBrotliParams) as type C.CBrotliParams in function argument go/src/gopkg.in/kothar/brotli-go.v0/enc/encode.go:170: cannot use params.c (type C.struct_CBrotliParams) as type C.CBrotliParams in function argument go/src/gopkg.in/kothar/brotli-go.v0/enc/encode.go:188: cannot use params.c (type C.struct_CBrotliParams) as type C.CBrotliParams in function argument
i have installed build-essentials and gcc. i also get this error on any other go project that uses brotli, like
https://github.com/barnacs/compy
With
$ go version
go version go1.6rc2 windows/amd64
...running tests fail:
=== RUN TestStreamDecompression
--- FAIL: TestStreamDecompression (0.05s)
panic: runtime error: cgo argument has Go pointer to Go pointer [recovered]
panic: runtime error: cgo argument has Go pointer to Go pointer
goroutine 5 [running]:
testing.tRunner.func1(0xc082076000)
C:/Go/src/testing/testing.go:467 +0x199
gopkg.in/kothar/brotli-go.v0/dec.NewBrotliReaderSize(0xb8a6b0, 0xc082007f80, 0x20000, 0xb8a6b0)
C:/msys64/home/amwenger/Dev/go/src/gopkg.in/kothar/brotli-go.v0/dec/decode.go:208 +0x130
gopkg.in/kothar/brotli-go.v0/dec.NewBrotliReader(0xb8a6b0, 0xc082007f80, 0x758ff0)
C:/msys64/home/amwenger/Dev/go/src/gopkg.in/kothar/brotli-go.v0/dec/decode.go:195 +0x3b
gopkg.in/kothar/brotli-go.v0/dec.TestStreamDecompression(0xc082076000)
C:/msys64/home/amwenger/Dev/go/src/gopkg.in/kothar/brotli-go.v0/dec/decode_test.go:28 +0x3c5
testing.tRunner(0xc082076000, 0x6ba5c0)
C:/Go/src/testing/testing.go:473 +0x9f
created by testing.RunTests
C:/Go/src/testing/testing.go:582 +0x899
FAIL gopkg.in/kothar/brotli-go.v0/dec 0.140s
I'm starting to think that we should wrap the brotli interface into something sane, ie. something that returns number of bytes consumed, etc. rather than moving around a bunch of pointers.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.