ogórek is a Go library for encoding and decoding pickles.
License: MIT License
ogórek is a Go library for encoding and decoding pickles.
Fuzz testing has been implemented for decoder and encoder. To run fuzz tests do the following:
go get github.com/dvyukov/go-fuzz/go-fuzz
go get github.com/dvyukov/go-fuzz/go-fuzz-build
go-fuzz-build github.com/kisielk/og-rek
go-fuzz -bin=./ogórek-fuzz.zip -workdir=./fuzz
Hi Kamil,
first of all. thank you for the package!
I got a problem with an attached pickle, it would be great if is solved.
I am trying to read tracer (https://github.com/Teichlab/tracer) output.
The file is attached. I get the following:
Error decoding : Unknown opcode 98 (b) at position 455: 'b'
Go version 1.9.* and the result is the same in Win64 and in openSUSE.
Actually, it is a strange pickle,
I cannot read it from a python script anywhere outside tracer's docker container.
The python3 inside the container reads it with usual
fname='test.pkl'
with open(fname,"rb") as f:
data=pickle.load(f)
Thank you!
Cheers,
Sasha.
I'm not sure if this really makes sense, but.... It sure would be handy if there were some mechanism by which pickled tuples could be decoded directly into structs. Having looked at xdrlib I think it could be done using reflection, but I'm still a bit green at golang to try and implement it myself. I suppose a list of names would have to be provided in the case that the tuple had multiple values of a given type.
https://github.com/davecgh/go-xdr/blob/master/xdr/decode.go#L32-L101
The escapes there are not handled. Found via fuzzing (#47):
"V\b\n."
panic: protocol 0: decode·encode != identity:
have: "\\b"
want: "\b"
goroutine 1 [running]:
github.com/kisielk/og-rek.Fuzz(0x7f8aab2f1000, 0x5, 0x200000, 0xc000034748)
/tmp/go-fuzz-build697921479/gopath/src/github.com/kisielk/og-rek/fuzz.go:87 +0x604
go-fuzz-dep.Main(0x524d78)
/tmp/go-fuzz-build697921479/goroot/src/go-fuzz-dep/main.go:49 +0xad
main.main()
/tmp/go-fuzz-build697921479/gopath/src/github.com/kisielk/og-rek/go.fuzz.main/main.go:10 +0x2d
exit status 2
similar for S.
We actually have a TODO for this in the code:
https://github.com/kisielk/og-rek/blob/v1.0.0-29-ge7d9696/ogorek.go#L604
But silently decoding to wrong data is frightening.
"((dd"
panic: runtime error: hash of unhashable type map[interface {}]interface {}
goroutine 1 [running]:
github.com/kisielk/og-rek.(*Decoder).loadDict(0xc420085170, 0x64, 0x0)
/tmp/go-fuzz-build403415384/gopath/src/github.com/kisielk/og-rek/ogorek.go:652 +0x18c
github.com/kisielk/og-rek.Decoder.Decode(0xc42001cd20, 0x5a9320, 0x0, 0x0, 0xc420016f00, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/tmp/go-fuzz-build403415384/gopath/src/github.com/kisielk/og-rek/ogorek.go:188 +0x172d
github.com/kisielk/og-rek.Fuzz(0x7f3e672e1000, 0x4, 0x200000, 0x3)
/tmp/go-fuzz-build403415384/gopath/src/github.com/kisielk/og-rek/fuzz.go:12 +0x108
go-fuzz-dep.Main(0x50d798)
/tmp/go-fuzz-build403415384/goroot/src/go-fuzz-dep/main.go:49 +0xd9
main.main()
/tmp/go-fuzz-build403415384/gopath/src/github.com/kisielk/og-rek/go.fuzz.main/main.go:10 +0x2d
exit status 2
"}}}s"
panic: runtime error: hash of unhashable type map[interface {}]interface {}
goroutine 1 [running]:
github.com/kisielk/og-rek.(*Decoder).loadSetItem(0xc420098000, 0xc42000a373, 0x0)
/tmp/go-fuzz-build403415384/gopath/src/github.com/kisielk/og-rek/ogorek.go:831 +0x13b
github.com/kisielk/og-rek.Decoder.Decode(0xc42001c120, 0x5a9320, 0x0, 0x0, 0xc4200161b0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/tmp/go-fuzz-build403415384/gopath/src/github.com/kisielk/og-rek/ogorek.go:220 +0x1a4c
github.com/kisielk/og-rek.Fuzz(0x7f0f87128000, 0x4, 0x200000, 0xc42003df78)
/tmp/go-fuzz-build403415384/gopath/src/github.com/kisielk/og-rek/fuzz.go:12 +0x108
go-fuzz-dep.Main(0x50d798)
/tmp/go-fuzz-build403415384/goroot/src/go-fuzz-dep/main.go:49 +0xd9
main.main()
/tmp/go-fuzz-build403415384/gopath/src/github.com/kisielk/og-rek/go.fuzz.main/main.go:10 +0x2d
exit status 2
"(((ld"
panic: runtime error: hash of unhashable type []interface {}
goroutine 1 [running]:
github.com/kisielk/og-rek.(*Decoder).loadDict(0xc4200894d0, 0x64, 0x0)
/tmp/go-fuzz-build403415384/gopath/src/github.com/kisielk/og-rek/ogorek.go:652 +0x18c
github.com/kisielk/og-rek.Decoder.Decode(0xc42001cf60, 0x5a9320, 0x0, 0x0, 0xc420016fc0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/tmp/go-fuzz-build403415384/gopath/src/github.com/kisielk/og-rek/ogorek.go:188 +0x172d
github.com/kisielk/og-rek.Fuzz(0x7fbfd4b96000, 0x5, 0x200000, 0x3)
/tmp/go-fuzz-build403415384/gopath/src/github.com/kisielk/og-rek/fuzz.go:12 +0x108
go-fuzz-dep.Main(0x50d798)
/tmp/go-fuzz-build403415384/goroot/src/go-fuzz-dep/main.go:49 +0xd9
main.main()
/tmp/go-fuzz-build403415384/gopath/src/github.com/kisielk/og-rek/go.fuzz.main/main.go:10 +0x2d
exit status 2
"}(U\x040000u"
panic: runtime error: index out of range
goroutine 1 [running]:
github.com/kisielk/og-rek.(*Decoder).loadSetItems(0xc420088000, 0x75, 0x0)
/tmp/go-fuzz-build403415384/gopath/src/github.com/kisielk/og-rek/ogorek.go:851 +0x250
github.com/kisielk/og-rek.Decoder.Decode(0xc42001c180, 0x5a9320, 0x0, 0x0, 0xc420016360, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/tmp/go-fuzz-build403415384/gopath/src/github.com/kisielk/og-rek/ogorek.go:232 +0x251d
github.com/kisielk/og-rek.Fuzz(0x7f466e803000, 0x9, 0x200000, 0xc42003df78)
/tmp/go-fuzz-build403415384/gopath/src/github.com/kisielk/og-rek/fuzz.go:12 +0x108
go-fuzz-dep.Main(0x50d798)
/tmp/go-fuzz-build403415384/goroot/src/go-fuzz-dep/main.go:49 +0xd9
main.main()
/tmp/go-fuzz-build403415384/gopath/src/github.com/kisielk/og-rek/go.fuzz.main/main.go:10 +0x2d
exit status 2
"S\n"
panic: runtime error: index out of range
goroutine 1 [running]:
github.com/kisielk/og-rek.(*Decoder).loadString(0xc4200882d0, 0xc420088253, 0x0)
/tmp/go-fuzz-build403415384/gopath/src/github.com/kisielk/og-rek/ogorek.go:502 +0x469
github.com/kisielk/og-rek.Decoder.Decode(0xc42001c360, 0x5a9320, 0x0, 0x0, 0xc4200164e0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/tmp/go-fuzz-build403415384/gopath/src/github.com/kisielk/og-rek/ogorek.go:172 +0x1199
github.com/kisielk/og-rek.Fuzz(0x7f5029057000, 0x2, 0x200000, 0x3)
/tmp/go-fuzz-build403415384/gopath/src/github.com/kisielk/og-rek/fuzz.go:12 +0x108
go-fuzz-dep.Main(0x50d798)
/tmp/go-fuzz-build403415384/goroot/src/go-fuzz-dep/main.go:49 +0xd9
main.main()
/tmp/go-fuzz-build403415384/gopath/src/github.com/kisielk/og-rek/go.fuzz.main/main.go:10 +0x2d
exit status 2
"(\x88d"
panic: runtime error: index out of range
goroutine 1 [running]:
github.com/kisielk/og-rek.(*Decoder).loadDict(0xc420082990, 0xc42000e464, 0x0)
/tmp/go-fuzz-build403415384/gopath/src/github.com/kisielk/og-rek/ogorek.go:652 +0x21d
github.com/kisielk/og-rek.Decoder.Decode(0xc42001c7e0, 0x5a9320, 0x0, 0x0, 0xc4200167b0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/tmp/go-fuzz-build403415384/gopath/src/github.com/kisielk/og-rek/ogorek.go:188 +0x172d
github.com/kisielk/og-rek.Fuzz(0x7f6d1f310000, 0x3, 0x200000, 0x3)
/tmp/go-fuzz-build403415384/gopath/src/github.com/kisielk/og-rek/fuzz.go:12 +0x108
go-fuzz-dep.Main(0x50d798)
/tmp/go-fuzz-build403415384/goroot/src/go-fuzz-dep/main.go:49 +0xd9
main.main()
/tmp/go-fuzz-build403415384/gopath/src/github.com/kisielk/og-rek/go.fuzz.main/main.go:10 +0x2d
exit status 2
e.g. found via fuzzing (#47):
"(\x88\x88l2a22aa."
runtime: goroutine stack exceeds 1000000000-byte limit
fatal error: stack overflow
runtime stack:
runtime.throw(0x51e8c6, 0xe)
/tmp/go-fuzz-build697921479/goroot/src/runtime/panic.go:608 +0x72
runtime.newstack()
/tmp/go-fuzz-build697921479/goroot/src/runtime/stack.go:1008 +0x729
runtime.morestack()
/tmp/go-fuzz-build697921479/goroot/src/runtime/asm_amd64.s:429 +0x8f
goroutine 1 [running]:
runtime.mallocgc(0x8, 0x0, 0x0, 0x0)
/tmp/go-fuzz-build697921479/goroot/src/runtime/malloc.go:773 +0x9ab fp=0xc023a34390 sp=0xc023a34388 pc=0x40b27b
runtime.rawbyteslice(0x4, 0x0, 0x0, 0x0)
/tmp/go-fuzz-build697921479/goroot/src/runtime/string.go:271 +0xa0 fp=0xc023a343d0 sp=0xc023a34390 pc=0x43fd10
runtime.stringtoslicebyte(0x0, 0x51d353, 0x4, 0x8, 0xc0004c9b30, 0x0)
/tmp/go-fuzz-build697921479/goroot/src/runtime/string.go:160 +0xb5 fp=0xc023a34418 sp=0xc023a343d0 pc=0x43f705
github.com/kisielk/og-rek.(*Encoder).emits(0xc043a33eb8, 0x51d353, 0x4, 0xc023a34490, 0x4924aa)
/tmp/go-fuzz-build697921479/gopath/src/github.com/kisielk/og-rek/encode.go:90 +0x5a fp=0xc023a34458 sp=0xc023a34418 pc=0x4d4efa
github.com/kisielk/og-rek.(*Encoder).encodeBool(0xc043a33eb8, 0x5da901, 0x81, 0x4f7101)
/tmp/go-fuzz-build697921479/gopath/src/github.com/kisielk/og-rek/encode.go:254 +0x19a fp=0xc023a344a0 sp=0xc023a34458 pc=0x4d66ba
github.com/kisielk/og-rek.(*Encoder).encode(0xc043a33eb8, 0x4f7160, 0x5da9a1, 0x81, 0x5da9a1, 0x81)
/tmp/go-fuzz-build697921479/gopath/src/github.com/kisielk/og-rek/encode.go:109 +0x1ff fp=0xc023a344f8 sp=0xc023a344a0 pc=0x4d529f
github.com/kisielk/og-rek.(*Encoder).encode(0xc043a33eb8, 0x4ff000, 0xc0001329c0, 0x194, 0x4ff000, 0xc0001329c0)
/tmp/go-fuzz-build697921479/gopath/src/github.com/kisielk/og-rek/encode.go:137 +0xa13 fp=0xc023a34550 sp=0xc023a344f8 pc=0x4d5ab3
github.com/kisielk/og-rek.(*Encoder).encodeArray(0xc043a33eb8, 0x4f5e80, 0xc0001329e0, 0x97, 0xc0001329e0, 0x4f5e80)
/tmp/go-fuzz-build697921479/gopath/src/github.com/kisielk/og-rek/encode.go:232 +0x1ee fp=0xc023a345a8 sp=0xc023a34550 pc=0x4d639e
github.com/kisielk/og-rek.(*Encoder).encode(0xc043a33eb8, 0x4f5e80, 0xc0001329e0, 0x97, 0xc0001329e0, 0x97)
/tmp/go-fuzz-build697921479/gopath/src/github.com/kisielk/og-rek/encode.go:122 +0x45e fp=0xc023a34600 sp=0xc023a345a8 pc=0x4d54fe
github.com/kisielk/og-rek.(*Encoder).encode(0xc043a33eb8, 0x4ff000, 0xc0001245a0, 0x194, 0x4ff000, 0xc0001245a0)
/tmp/go-fuzz-build697921479/gopath/src/github.com/kisielk/og-rek/encode.go:137 +0xa13 fp=0xc023a34658 sp=0xc023a34600 pc=0x4d5ab3
github.com/kisielk/og-rek.(*Encoder).encodeArray(0xc043a33eb8, 0x4f5e80, 0xc000132a20, 0x97, 0xc000132a20, 0x4f5e80)
/tmp/go-fuzz-build697921479/gopath/src/github.com/kisielk/og-rek/encode.go:232 +0x1ee fp=0xc023a346b0 sp=0xc023a34658 pc=0x4d639e
github.com/kisielk/og-rek.(*Encoder).encode(0xc043a33eb8, 0x4f5e80, 0xc000132a20, 0x97, 0xc000132a20, 0x97)
/tmp/go-fuzz-build697921479/gopath/src/github.com/kisielk/og-rek/encode.go:122 +0x45e fp=0xc023a34708 sp=0xc023a346b0 pc=0x4d54fe
github.com/kisielk/og-rek.(*Encoder).encode(0xc043a33eb8, 0x4ff000, 0xc0001245b0, 0x194, 0x4ff000, 0xc0001245b0)
/tmp/go-fuzz-build697921479/gopath/src/github.com/kisielk/og-rek/encode.go:137 +0xa13 fp=0xc023a34760 sp=0xc023a34708 pc=0x4d5ab3
...
It decodes self-referencing object on which the Encoder infinitely recurses (see ^^^):
In [5]: pickle.loads(s)
Out[5]: [True, True, [...], [...], [...]]
The pickle itself with details:
In [4]: dis(s)
0: ( MARK
1: \x88 NEWTRUE
2: \x88 NEWTRUE
3: l LIST (MARK at 0)
4: 2 DUP
5: a APPEND
6: 2 DUP
7: 2 DUP
8: a APPEND
9: a APPEND
10: . STOP
highest protocol among opcodes = 2
Currently we decode longs to *big.Int and it kind-of works. Also despite == not giving equality for two *big.Int numbers, reflect.DeepEqual works for two such numbers since the numbers are internally always stored in the same form. However not working == poses a problem:
*big.Int cannot be used as map key for lookup: the lookup will use == and lookup will succed only if accessing something with pointer to exactly the same *big.Int object that was used at map creation.reflect.DeepEqual for maps with *big.Int key(s) wont work - for maps reflect.DeepEqual compares deeply only values and keys are compared by regular Go ==.All this in turn leads to that some pickles with long keys are decoded wrongly, e.g.:
# python
In [9]: q = "}(L1L\nS'aaa'\nL1L\nS'bbb'\nu."
In [10]: dis(q)
0: } EMPTY_DICT
1: ( MARK
2: L LONG 1L
6: S STRING 'aaa'
13: L LONG 1L
17: S STRING 'bbb'
24: u SETITEMS (MARK at 1)
25: . STOP
highest protocol among opcodes = 1
In [11]: pickle.loads(q)
Out[11]: {1L: 'bbb'}
but ogórek gives:
map[interface {}]interface {}{1:"aaa", 1:"bbb"}
Probably nothing we can do here until in Go2 they make builtin int to be really a bigint (golang/go#19623), and then, similarly to Python3, we decode both INT and LONG into Go int, and on encoding Go int, we look into its range and if it is in int32 range -> go INT/BININT/..., and otherwise go via LONG*.
If tuple is used directly, for example:
In [11]: s = "((tNd."
In [19]: dis(s)
0: ( MARK
1: ( MARK
2: t TUPLE (MARK at 1)
3: N NONE
4: d DICT (MARK at 0)
5: . STOP
highest protocol among opcodes = 0
In [20]: loads(s)
Out[20]: {(): None}
our Decoder gives:
pickle: loadDict: invalid key type ogórek.Tuple
However the pickle is valid by Python rules because there tuple is immutable (and thus hashable).
However the check inside Decoder is not complete - for example if we hook Tuple into another type whos is comparable (e.g. Ref here):
In [11]: s = "((tQNd."
In [15]: dis(s)
0: ( MARK
1: ( MARK
2: t TUPLE (MARK at 1)
3: Q BINPERSID
4: N NONE
5: d DICT (MARK at 0)
6: . STOP
highest protocol among opcodes = 1
our Decoder crashes:
panic: runtime error: hash of unhashable type ogórek.Tuple
goroutine 1 [running]:
github.com/kisielk/og-rek.(*Decoder).loadDict(0xc00009efd0, 0xc00000a264, 0x0)
/tmp/go-fuzz-build914098789/gopath/src/github.com/kisielk/og-rek/ogorek.go:819 +0x32b
github.com/kisielk/og-rek.(*Decoder).Decode(0xc00009efd0, 0xc00009aa10, 0xc00009efd0, 0x464f39, 0x4)
/tmp/go-fuzz-build914098789/gopath/src/github.com/kisielk/og-rek/ogorek.go:242 +0x14fa
github.com/kisielk/og-rek.Fuzz(0x7f62339cc000, 0x6, 0x200000, 0x3)
/tmp/go-fuzz-build914098789/gopath/src/github.com/kisielk/og-rek/fuzz.go:15 +0xdf
go-fuzz-dep.Main(0x524df8)
/tmp/go-fuzz-build914098789/goroot/src/go-fuzz-dep/main.go:49 +0xad
main.main()
/tmp/go-fuzz-build914098789/gopath/src/github.com/kisielk/og-rek/go.fuzz.main/main.go:10 +0x2d
exit status 2
package main
import (
"bytes"
"encoding/binary"
"fmt"
pickle "github.com/kisielk/og-rek"
pickle2 "github.com/lomik/og-rek"
"math"
"time"
)
// Fake single interval set for graphite
type IntervalSet struct {
Start int32
End int32
}
func (i *IntervalSet) MarshalPickle() ([]byte, error) {
// 0: ( MARK
// 1: c GLOBAL 'graphite.intervals IntervalSet'
// 33: o OBJ (MARK at 0)
// 34: } EMPTY_DICT
// 35: ( MARK
// 36: U SHORT_BINSTRING 'intervals'
// 47: ] EMPTY_LIST
// 48: ( MARK
// 49: c GLOBAL 'graphite.intervals Interval'
// 78: o OBJ (MARK at 48)
// 79: } EMPTY_DICT
// 80: ( MARK
// 81: U SHORT_BINSTRING 'start'
// 88: G BINFLOAT 1322087998.393128
// 97: U SHORT_BINSTRING 'size'
// 103: G BINFLOAT 157679977.1475761
// 112: U SHORT_BINSTRING 'end'
// 117: G BINFLOAT 1479767975.540704
// 126: U SHORT_BINSTRING 'tuple'
// 133: G BINFLOAT 1322087998.393128
// 142: G BINFLOAT 1479767975.540704
// 151: \x86 TUPLE2
// 152: u SETITEMS (MARK at 80)
// 153: b BUILD
// 154: a APPEND
// 155: U SHORT_BINSTRING 'size'
// 161: G BINFLOAT 157679977.1475761
// 170: u SETITEMS (MARK at 35)
// 171: b BUILD
// 172: . STOP
b := []byte("(cgraphite.intervals\nIntervalSet\no}(U\tintervals](cgraphite.intervals\nInterval\no}(U\x05startGA\xd3\xb3]\x8f\x99)\x02U\x04sizeGA\xa2\xcc\x02\xd2K\x8f\x18U\x03endGA\xd6\x0c\xdd\xe9\xe2\x9a\xe5U\x05tupleGA\xd3\xb3]\x8f\x99)\x02GA\xd6\x0c\xdd\xe9\xe2\x9a\xe5\x86ubaU\x04sizeGA\xa2\xcc\x02\xd2K\x8f\x18ub")
binary.BigEndian.PutUint64(b[89:97], uint64(math.Float64bits(float64(i.Start))))
binary.BigEndian.PutUint64(b[104:112], uint64(math.Float64bits(float64(i.End-i.Start))))
binary.BigEndian.PutUint64(b[118:126], uint64(math.Float64bits(float64(i.End))))
binary.BigEndian.PutUint64(b[134:142], uint64(math.Float64bits(float64(i.Start))))
binary.BigEndian.PutUint64(b[143:151], uint64(math.Float64bits(float64(i.End))))
binary.BigEndian.PutUint64(b[162:170], uint64(math.Float64bits(float64(i.End-i.Start))))
return b, nil
}
func encodePickleold(data interface{}) ([]byte, error) {
var buf bytes.Buffer
penc := pickle2.NewEncoder(&buf)
err := penc.Encode(data)
if err != nil {
fmt.Println("err:", err)
return nil, err
}
return buf.Bytes(), nil
}
func encodePickle(protocol int,data interface{}) ([]byte, error) {
var buf bytes.Buffer
penc := pickle.NewEncoderWithConfig(&buf, &pickle.EncoderConfig{
// allow both Python2 and Python3 to decode what ogórek produces by default
Protocol: protocol,
})
err := penc.Encode(data)
if err != nil {
fmt.Println("err:", err)
return nil, err
}
return buf.Bytes(), nil
}
func decodePickle(data []byte) (interface{}, error) {
d := pickle.NewDecoder(bytes.NewReader(data))
output, err := d.Decode()
//if err != nil {
// beego.Error("err:", err)
//}
//beego.Info("[TEST]output:", output)
return output, err
}
func main() {
value := make(map[string]interface{}, 0)
value["is_leaf"] = false
value["path"] = "stats"
value["intervals"] = &IntervalSet{1445184000, int32(time.Now().Unix())}
for i:=0 ;i< 5;i++ {
by,err := encodePickle(i,value)
if err != nil {
fmt.Println("err:", err)
}
fmt.Println("by",i ,":",by)
}
byOld,errOld := encodePickleold(value)
if errOld != nil {
fmt.Println("errOld:", errOld)
}
fmt.Println("byOld:", byOld)
}
But, only data of "byOld,errOld := encodePickleold(value)" can be identified by "github.com/graphite-project/graphite-web"
Hello,
are there any plans on supporting Pickle Protocol Version 5 which was introduced with Python 3.8?
https://docs.python.org/3/library/pickle.html#pickle-protocols
Unfortunately, I have no clue about go language, I just need this as a dependency...
Thanks for your help!
1、can not decode The following code in 'go',but 'python' can decode
package main
import (
"bytes"
"encoding/binary"
"fmt"
pickle "github.com/lomik/og-rek"
"math"
"reflect"
"time"
)
// Fake single interval set for graphite
type IntervalSet struct {
Start int32
End int32
}
func (i *IntervalSet) MarshalPickle() ([]byte, error) {
// 0: ( MARK
// 1: c GLOBAL 'graphite.intervals IntervalSet'
// 33: o OBJ (MARK at 0)
// 34: } EMPTY_DICT
// 35: ( MARK
// 36: U SHORT_BINSTRING 'intervals'
// 47: ] EMPTY_LIST
// 48: ( MARK
// 49: c GLOBAL 'graphite.intervals Interval'
// 78: o OBJ (MARK at 48)
// 79: } EMPTY_DICT
// 80: ( MARK
// 81: U SHORT_BINSTRING 'start'
// 88: G BINFLOAT 1322087998.393128
// 97: U SHORT_BINSTRING 'size'
// 103: G BINFLOAT 157679977.1475761
// 112: U SHORT_BINSTRING 'end'
// 117: G BINFLOAT 1479767975.540704
// 126: U SHORT_BINSTRING 'tuple'
// 133: G BINFLOAT 1322087998.393128
// 142: G BINFLOAT 1479767975.540704
// 151: \x86 TUPLE2
// 152: u SETITEMS (MARK at 80)
// 153: b BUILD
// 154: a APPEND
// 155: U SHORT_BINSTRING 'size'
// 161: G BINFLOAT 157679977.1475761
// 170: u SETITEMS (MARK at 35)
// 171: b BUILD
// 172: . STOP
b := []byte("(cgraphite.intervals\nIntervalSet\no}(U\tintervals](cgraphite.intervals\nInterval\no}(U\x05startGA\xd3\xb3]\x8f\x99)\x02U\x04sizeGA\xa2\xcc\x02\xd2K\x8f\x18U\x03endGA\xd6\x0c\xdd\xe9\xe2\x9a\xe5U\x05tupleGA\xd3\xb3]\x8f\x99)\x02GA\xd6\x0c\xdd\xe9\xe2\x9a\xe5\x86ubaU\x04sizeGA\xa2\xcc\x02\xd2K\x8f\x18ub")
binary.BigEndian.PutUint64(b[89:97], uint64(math.Float64bits(float64(i.Start))))
binary.BigEndian.PutUint64(b[104:112], uint64(math.Float64bits(float64(i.End-i.Start))))
binary.BigEndian.PutUint64(b[118:126], uint64(math.Float64bits(float64(i.End))))
binary.BigEndian.PutUint64(b[134:142], uint64(math.Float64bits(float64(i.Start))))
binary.BigEndian.PutUint64(b[143:151], uint64(math.Float64bits(float64(i.End))))
binary.BigEndian.PutUint64(b[162:170], uint64(math.Float64bits(float64(i.End-i.Start))))
return b, nil
}
func encodePickle(data interface{}) ([]byte, error) {
var buf bytes.Buffer
fmt.Println("type:", reflect.TypeOf(data))
penc := pickle.NewEncoder(&buf)
err := penc.Encode(data)
if err != nil {
println("encodePickle err:", err)
return nil, err
}
return buf.Bytes(), nil
}
func decodePickle(data []byte) (interface{}, error) {
d := pickle.NewDecoder(bytes.NewReader(data))
output, err := d.Decode()
if err != nil {
println("decodePickle err:", err)
}
return output, err
}
func main() {
value := make(map[string]interface{}, 0)
value["is_leaf"] = false
value["path"] = "stats"
value["intervals"] = &IntervalSet{1445184000, int32(time.Now().Unix())}
by,err3 := encodePickle(value)
if err3 != nil {
fmt.Println("err3:", err3)
}
fmt.Println("by:", by)
result, err4 := decodePickle(by)
if err4 != nil {
fmt.Println("err4:", err4)
}else {
fmt.Println("result:", result)
}
}
2、
If changed to following code can run, but I have to use ‘&’
value["intervals"] = IntervalSet{1445184000, int32(time.Now().Unix())}
With this opcode there will never be a non-lossy serialization, but do you think we could do something that could get us half way there?
http://hg.python.org/cpython/file/9c724c428e1f/Lib/pickle.py#l1320
Could we push a struct that would have the same module/class-name onto the stack? (in place of Python's #'find_class)
If you think that implementing this opcode would do more harm than good I totally understand. I am open to any ideas, and would love to work on it myself if you don't want to work on it.
The filenames for ogórek.go and ogórek_test.go use the Latin Small Letter O + the Combining Acute Accent to compose the equivalent Latin Small Letter O With Acute. This causes issues with various bits of tooling on Windows systems, including go get (though I think the actual issue lies somewhere within the Git client for Windows, probably some piece of its tooling), where the filename isn't recognized as valid.
Switching the filenames to use the pre-composed Latin Small Letter O With Acute (which, in an odd twist, is what's used internally for the the ogórek package name) solves the issue.
I realize that the actual problem here is that the affected Windows tooling isn't supporting the combining characters as they should, however, changing the filenames to use the pre-composed character is an easy and quick fix.
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.