How to reproduce:

mkdir tmp

cd tmp

cat >foo.go <<EOF
package foo
import _ "github.com/ugorji/go/codec"
EOF

go get

GOARCH=386 go build
# github.com/ugorji/go/codec
../src/github.com/ugorji/go/codec/binc.go:279: constant 4294967295 overflows int

To emphasize importance of this issue: it breaks (at least local) Appengine builds.
ERROR 2013-06-07 08:52:54,896 go_runtime.py:165] Failed to build Go application: /home/temoto/src/github.com/ugorji/go/codec/binc.go:279: constant 4294967295 overflows int

Not strictly an issue, but is there anywhere a working example of how to encode/ deconde from start to end? Really does not matter into the details, but I can't figure out how one it's even supposed to start.
It's most probably me being idiot but I can't figure out from the example in the readme, which to my newbie eyes has some obscure things.

:D

Latest version is causing issues for us, as we now get the following:

/opt/gopath/src/github.com/ugorji/go/codec/helper.go:504 (0x51c024)
    com/ugorji/go/codec.panicToErr: debug.PrintStack()
/opt/go/src/pkg/runtime/panic.c:229 (0x413f61)
    panic: reflect·call(d->fn, (byte*)d->args, d->siz);
/opt/go/src/pkg/runtime/iface.c:564 (0x40ae58)
    ifacehash1: runtime·panic(err);
/opt/go/src/pkg/runtime/iface.c:584 (0x40af59)
    efacehash: return ifacehash1(a.data, a.type, h);
/opt/go/src/pkg/runtime/alg.c:387 (0x4032eb)
    nilinterhash: *h = runtime·efacehash(*(Eface*)a, *h ^ M0) * M1;
/opt/go/src/pkg/runtime/hashmap.c:597 (0x40834d)
    hash_insert: t->key->alg->hash(&hash, t->key->size, key);
/opt/go/src/pkg/runtime/hashmap.c:1272 (0x4099d7)
    mapassign: hash_insert(t, h, ak, av);
/opt/go/src/pkg/runtime/hashmap.c:1301 (0x409a5d)
    mapassign1: runtime·mapassign(t, h, ak, av);
/opt/gopath/src/github.com/ugorji/go/codec/decode.go:904 (0x51563f)
    com/ugorji/go/codec.(*Decoder).decMapIntfIntf: m[mk] = mv
/opt/gopath/src/github.com/ugorji/go/codec/decode.go:682 (0x5141ff)
    com/ugorji/go/codec.(*Decoder).decode: d.decMapIntfIntf(v)
/opt/gopath/src/github.com/ugorji/go/codec/decode.go:341 (0x511b2b)
    com/ugorji/go/codec.(*decFnInfo).kInterface: f.d.decode(v)
/opt/gopath/src/github.com/ugorji/go/codec/decode.go:817 (0x51496a)
    com/ugorji/go/codec.(*Decoder).decodeValue: fn.f(fn.i, rv)
/opt/gopath/src/github.com/ugorji/go/codec/decode.go:687 (0x51402f)
    com/ugorji/go/codec.(*Decoder).decode: d.decodeValue(reflect.ValueOf(iv).Elem())
/opt/gopath/src/github.com/ugorji/go/codec/decode.go:626 (0x5135e7)
    com/ugorji/go/codec.(*Decoder).Decode: d.decode(v)
/opt/gopath/src/github.com/ugorji/go/codec/rpc.go:79 (0x51fbc1)
    com/ugorji/go/codec.(*rpcCodec).read: return c.dec.Decode(obj)
/opt/gopath/src/github.com/ugorji/go/codec/rpc.go:87 (0x51fc69)
    com/ugorji/go/codec.(*rpcCodec).ReadResponseBody: return c.read(body)
/opt/go/src/pkg/net/rpc/client.go:138 (0x4ac74e)
    (*Client).input: err = client.codec.ReadResponseBody(call.Reply)
/opt/go/src/pkg/runtime/proc.c:1223 (0x417ae0)
    goexit: runtime·goexit(void)
2013/10/17 00:52:38 rpc: client protocol error: runtime error: hash of unhashable type []uint8

Still trying to get a minimal test case, but I can reproduce this in our project test suite.

The fix to https://github.com/ugorji/go-msgpack/issues/10 that was carried over into the codec implementation was incorrect.

The 4th parameter to msgpack rpc is an array of all the arguments. However, Go requires that methods exposed through net/rpc have only a single argument. By having the "extra" array indirection on read and write, the "single argument" vs "array of arguments" problem is solved by passing an array consisting of a single argument, and both Go and msgpack are happy. This means that when using msgpack-rpc between Go and (say) Python, the more-restrictive calling conventions of Go (namely, only a single parameter) must be respected. The rpcCodec.ReadRequestBody routine can't be shared, it must be specialized. msgpackSpecRpcCodec writeCustomBody just needs to be patched.

You can test this against these trimmed-down client/servers in https://github.com/dgryski/trifles/tree/master/msgrpc

I use Go 1.4.1.

MsgpackDecoder produce int64 for unsigned integer while I set SignedInteger = false.

https://gist.github.com/methane/cfdcce8257a6406dd2ad
Output of this code is 42, int64.

Hi, I often use your great library, thanks.

I encountered a failure in decoding long []byte with BincHandle.
The following code is an example to reproduce the failure.
I expect that the both lengths should be 65536, but the length of decoded one is 256.

package main

import (
    "fmt"
    "github.com/ugorji/go/codec"
    "os"
)

func main() {
    b := make([]byte, 0)
    var h codec.BincHandle
    enc := codec.NewEncoderBytes(&b, &h)
    sample := make([]byte, 65536)
    err := enc.Encode(sample)
    if err != nil {
        fmt.Println(err)
        os.Exit(1)
    }

    dec := codec.NewDecoderBytes(b, &h)
    sampleDecoded := make([]byte, 0)
    err = dec.Decode(&sample_test)
    if err != nil {
        fmt.Println(err)
        os.Exit(1)
    }

    fmt.Printf("len=%d\n", len(sample))
    fmt.Printf("len=%d\n", len(sampleDecoded)
}

Hi, I've been looking at codec recently and was testing it out but was getting some behavior differences between it and encoding/json. Here is the output I'm getting using the same structs.

Encodings from codec generated selfer code....
{"Key":"thing"}
{"Key":"whynot","Field1":"another"}
Encodings from codec reflection based encoding....
{"_key":"thing"}
{"_id":"another","_key":"whynot"}
Encodings from encoding/json....
{"_key":"thing"}
{"_key":"whynot","_id":"another"}

Below is my example code and directory structure

root
|----main.go
|----model
|--------model.go

The main.go file

package main

import (
    "fmt"
    "github.com/ugorji/go/codec" 
    "encoding/json"
    "./model"
)

type Test struct {
    Key string `json:"_key,omitempty"`
}

type Next struct {
    Test
    Field1 string `json:"_id,omitempty"`
    Field2 string `json:",omitempty"`
}


func main() {
    var h codec.Handle = new(codec.JsonHandle)
    var b []byte

    d := &model.Test{ Key : "thing" }
    u := &model.Next{ Field1: "another" }

    f := &Test{ Key : "thing" }
    t := &Next{ Field1: "another" }

    u.Key = "whynot"
    t.Key = "whynot" //Edited to add this line since it was missing

    fmt.Println( "Encodings from codec generated selfer code...." )
    b, _ = encodeCodec( d, h )
    fmt.Println( string(b) )
    b, _ = encodeCodec( u, h )
    fmt.Println( string(b) )

    fmt.Println( "Encodings from codec reflection based encoding...." )
    b, _ = encodeCodec( f, h )
    fmt.Println( string(b) )
    b, _ = encodeCodec( t, h )
    fmt.Println( string(b) )

    fmt.Println( "Encodings from encoding/json...." )
    b, _ = encodeJson( d )
    fmt.Println( string(b) )
    b, _ = encodeJson( u )
    fmt.Println( string(b) )

}

func encodeCodec( v interface{}, h codec.Handle ) ( []byte, error ) {
    var b []byte
    enc := codec.NewEncoderBytes( &b, h )
    err := enc.Encode( v )
    if err != nil {
        fmt.Println( err )
        return nil, err
    }
    return b, nil
}

func encodeJson( v interface{} ) ( []byte, error ){
    b, err := json.Marshal( v )

    if err != nil {
        fmt.Println( err )
        return nil, err
    }
    return b, nil
}

Here is the model.go file

package model
//go:generate codecgen -o codec_gen.go model.go

type Test struct {
    Key string `json:"_key,omitempty"`
}

type Next struct {
    Test
    Field1 string `json:"_id,omitempty"`
    Field2 string `json:",omitempty"`
}

i have a file, with a batch of records, each record is a map with string key and string/int/float value. then i decode this file with code below:

type MsgpackReader struct {
    fp  *os.File
    dec *codec.Decoder
}

func NewMsgpackReader(filename string) *MsgpackReader {
    fp, err := os.Open(filename)
    if err != nil {
        return nil
    }
    r := bufio.NewReader(fp)
    mh := codec.MsgpackHandle{}
    mh.MapType = reflect.TypeOf(map[string]interface{}(nil))
    mh.RawToString = true
    return &MsgpackReader{fp: fp, dec: codec.NewDecoder(r, &mh)}
}

func (reader *MsgpackReader) Read() (record TRecord, err error) {
    err = reader.dec.Decode(&record)
    return
}

func (reader *MsgpackReader) Close() {
    reader.fp.Close()
}

func main() {
    reader := NewMsgpackReader(filename)
    defer reader.Close()
    for {
        _, err := reader.Read()
        if err != nil {
            if err == io.EOF {
                break
            } else {
                continue
            }
        }
    }
}

this file contains 874589 records, is there any wrong? i am confused.

Hi,

I am trying to encode a struct that contains a map[string]<comparable struct> (which is still making the whole struct incomparable). I am getting

panic: runtime error: comparing uncomparable type data.App

goroutine 13 [running]:
runtime.panic(0x40b540, 0xc21007aa50)
    /usr/local/go/src/pkg/runtime/panic.c:266 +0xb6
github.com/ugorji/go/codec.isEmptyValue(0x4596a0, 0xc210051750, 0x192, 0x0)
    /Users/tchap/Projects/Go/src/github.com/ugorji/go/codec/helper_internal.go:60 +0x34d
github.com/ugorji/go/codec.(*encFnInfo).kStruct(0xc2100b8450, 0x4133a0, 0xc210051750, 0x192)
    /Users/tchap/Projects/Go/src/github.com/ugorji/go/codec/encode.go:472 +0x27c
github.com/ugorji/go/codec.(*Encoder).encodeValue(0xc2100abcb0, 0x4133a0, 0xc210051750, 0x192)
    /Users/tchap/Projects/Go/src/github.com/ugorji/go/codec/encode.go:893 +0x45a
github.com/ugorji/go/codec.(*Encoder).encode(0xc2100abcb0, 0x4133a0, 0xc210051750)
    /Users/tchap/Projects/Go/src/github.com/ugorji/go/codec/encode.go:808 +0x1f1
github.com/ugorji/go/codec.(*Encoder).Encode(0xc2100abcb0, 0x4133a0, 0xc210051750, 0x0, 0x0)
    /Users/tchap/Projects/Go/src/github.com/ugorji/go/codec/encode.go:692 +0x69

where

type App struct {
    Id          bson.ObjectId `bson:"_id"            codec:"-"`
    Alias       string        `bson:"alias"          codec:"alias"`
    Name        string        `bson:"name"           codec:"name,omitempty"`
    Version     string        `bson:"version"        codec:"version,omitempty"`
    Description string        `bson:"description"    codec:"description,omitempty"`
    Repository  string        `bson:"repository"     codec:"repository,omitempty"`
    Vars        Variables     `bson:"vars,omitempty" codec:"vars,omitempty"        json:"variables"`
    Enabled     bool          `bson:"enabled"        codec:"enabled,omitempty"`
    Status      string        `bson:"-"              codec:"status,omitempty"`
}

type Variables map[string]Variable

type Variable struct {
    Usage    string `bson:"usage"              codec:"usage"`
    Type     string `bson:"type"               codec:"type"`
    Secret   bool   `bson:"secret,omitempty"   codec:"secret,omitempty"`
    Optional bool   `bson:"optional,omitempty" codec:"optional,omitempty"`
    Value    string `bson:"value"              codec:"value"`
}

The new changes to fast-path slices and maps has caused nil values to be replaced with non-nil zero values of the fields. This has caused lots of pain for us as the input is no longer equal to the output when serializing / deserializing. We've forked and rolled back to 71c2886

Running codecgen results in the following output:

func init() {
if codec1978.GenVersion != 1 {
_, file, _, _ := runtime.Caller(0)
err := fmt.Errorf("codecgen version mismatch: current: %v, need %v. Re-generate file: %v", 
1, codec1978.GenVersion, file)
panic(err)
}
if false { // reference the types, but skip this branch at build/run time
 = } 
}

The stray = on the penultimate line is incorrect and stops compilation. Removing this seems to result in correct code.

package main

import (
    "fmt"
    "github.com/ugorji/go/codec"
    "time"
)

func main() {
    bincHandle := new(codec.BincHandle)
    var b []byte
    t := time.Now()
    if e := codec.NewEncoderBytes(&b, bincHandle).Encode(t); e != nil {
        panic(e)
    }
    var t2 time.Time
    if e := codec.NewDecoderBytes(b, bincHandle).Decode(&t2); e != nil {
        panic(e)
    }
    fmt.Printf("%v became %v\n", t, t2)
    t = time.Time{}
    if e := codec.NewEncoderBytes(&b, bincHandle).Encode(t); e != nil {
        panic(e)
    }
    if e := codec.NewDecoderBytes(b, bincHandle).Decode(&t2); e != nil {
        panic(e)
    }
    fmt.Printf("%v became %v\n", t, t2)
}

produces

[0:zond@pelican ~]$go run tmp/test.go
2013-08-02 15:25:53.082092351 +0200 CEST became 2013-08-02 15:25:53.082092351 +0200 UTC+02:00
0001-01-01 00:00:00 +0000 UTC became 34843-02-20 00:36:16 +0000 UTC
[0:zond@pelican ~]$

I did some benchmarks of RPC encoding speeds.

https://gist.github.com/pcdummy/6168792

Do you know why Binc and MsgPack perform that bad?

Trying to get a struct encoded and cannot get this to work.

Haven't found any definitive examples that illustrate encoding structs.

Any ideas why this always returns an empty map?

package main

import (
    "bytes"
    "log"

    "github.com/ugorji/go/codec"
)

type status struct {
    rssi uint8  `codec:"rssi"`
    lqi  uint8  `codec:"lqi"`
    ts   uint16 `codec:"ts"`
}

func main() {
    in := status{
        rssi: 12,
        lqi:  231,
        ts:   12312,
    }
    //in := map[string]interface{}{"rssi": uint8(12), "lqi": uint8(231), "ts": uint16(12312)}

    w := new(bytes.Buffer)
    enc := codec.NewEncoder(w,
        &codec.MsgpackHandle{RawToString: true, WriteExt: true})

    _ = enc.Encode(in)

    log.Printf("% x", w.Bytes())

}

Here is the code I am using to encode/decode the maps. For my tests I am passing in a map with string key and value. The decoded map contains string key but []uint8 as value. I expect it to return the same map structure which was encoded.

Enocde / Decode code

func encode(msg map[string]interface{}) ([]byte, error) {
    var encoded []byte
    enc := codec.NewEncoderBytes(&encoded, &mh)
    err := enc.Encode(msg)
    if err != nil {
        return nil, err
    }

    return encoded, nil
}

func decode(encoded []byte) (map[string]interface{}, error) {
    var decoded map[string]interface{}
    dec := codec.NewDecoderBytes(encoded, &mh)
    err := dec.Decode(&decoded)
    if err != nil {
        return nil, err
    }
    return decoded, nil
}

Test

import (
    . "gopkg.in/check.v1"
    "testing"
)

// Hook up gocheck into the "go test" runner.
func Test(t *testing.T) {
    TestingT(t)
}

type S struct{}
var _ = Suite(&S{})
func (s *S) TestEncodeDecode(c *C) {
    msg := make(map[string]interface{})
    msg["Hello"] = string("world")
    encoded, err := encode(msg)
    c.Assert(err, IsNil)
    c.Assert(encoded, NotNil)
    decoded, err := decode(encoded)
    c.Assert(err, IsNil)
    c.Assert(decoded, DeepEquals, msg)
}

Here is the output of the test run.

FAIL: test.go:27: S.TestEncodeDecode

test.go:35:
c.Assert(decoded, DeepEquals, msg) ... obtained map[string]interface {} = map[string]interface
{}{"Hello":[]uint8{0x77, 0x6f, 0x72, 0x6c, 0x64}} ... expected
map[string]interface {} = map[string]interface {}{"Hello":"world"}

OOPS: 0 passed, 1 FAILED
--- FAIL: Test (0.00 seconds) FAIL

Example code

    var mh2 codec.MsgpackHandle
    ooo := []byte{
        220, 0, 117, //0xdc array 16, len 117
        204, 131, 204, 169, 112, 114, 111, 102, 105, 108,
        101, 105, 100, 204, 169, 48, 58, 48, 58, 49,
        48, 48, 48, 49, 204, 177, 97, 116, 116, 114,
        105, 98, 117, 116, 101, 115, 75, 101, 121, 76,
        105, 115, 116, 204, 147, 204, 164, 110, 97, 109,
        101, 204, 167, 115, 97, 118, 101, 118, 101, 114,
        204, 163, 108, 118, 108, 204, 171, 112, 97, 115,
        115, 116, 104, 114, 111, 117, 103, 104, 204, 217,
        36, 49, 49, 51, 100, 55, 102, 51, 52, 45,
        54, 100, 48, 97, 45, 52, 48, 55, 57, 45,
        98, 51, 50, 53, 45, 98, 101, 51, 50, 101,
        52, 98, 51, 101, 48, 97, 97,
    }
    dec4 := codec.NewDecoderBytes(ooo, &mh2)
    var abcn []byte
    dec4.Decode(&abcn)
    fmt.Println("---%v", abcn)

Output abcn is empty.

It appears that arrays are not able to be serialized.

package main

import (
    "github.com/ugorji/go/codec"
    "bytes"
    "log"
)

type bstruct struct {
    B [3]byte
}

func main() {
    b := bstruct{
        [3]byte{5, 7, 8},
    }

    log.Println(b)

    var m codec.MsgpackHandle

    var buf bytes.Buffer
    encoder := codec.NewEncoder(&buf, &m)
    if err := encoder.Encode(b); err != nil {
        log.Fatal("Failed to encode:", err)
    }
}

This snippet produces:
2013/11/14 09:01:41 Failed to encode:reflect.Value.Slice: slice of unaddressable array

Are arrays not able to be serialized?

The following code:

package main

import (
    "fmt"
    "github.com/zond/go/codec"
)

func main() {
    bincHandle := new(codec.BincHandle)
    var b []byte
    sl := make([]string, 0)
    if e := codec.NewEncoderBytes(&b, bincHandle).Encode(sl); e != nil {
        panic(e)
    }
    var sl2 []string
    if e := codec.NewDecoderBytes(b, bincHandle).Decode(&sl); e != nil {
        panic(e)
    }
    fmt.Printf("%#v became %#v\n", sl, sl2)
}

Produces:

[]string{} became []string(nil)

Empty slices are not always nil..

$ go get github.com/ugorji/go/codec
# github.com/ugorji/go/codec
../../ugorji/go/codec/encode.go:514: undefined: sync.Pool
../../ugorji/go/codec/encode.go:1177: undefined: sync.Pool

Should we pull from a different repos? Thanks!

When i use json decode codec, can i use something like:

type A struct {
        Hello float64 `json:"hello"`
        World string  `json:"world"`
    }

?

Currently pointer types are not valid for declaring extended types. This can create problems when the desired output is to set an interface value with a pointer value. To solve this problem for my case I simply made all decoding into interfaces as pointers, but I think a more general solution is needed.

My solution (still break unit tests) dmcgowan@5d26f5f

It would be nice if the import path mapped directly to a URL to the source code.

Currently, it doesn't, and for no good reason. Initially, we used a repository name of go, with the
hope of putting multiple packages into the same repository. However, in retrospect, that is not a
good idea; one repository should only hold one package, with possible sub-packages which relate
directly to that package.

The plan of action is:

• Use github to change repository name from go to go-codec
• Move all files from the codec sub-directory of the repository to the top-level directory
• Keep the codec subdirectory there, and create a codec/fail.go file

The new codec/fail.go file will contain

• package docs that tell users the new import path and where to find docs
• init method that panics with an appropriate error message
• possibly use the // import label to allow go get fail early

We hope to do this soon, likely by the end of january.

Can you please explain how to migrate this code:

var MpackHandle codec.MsgpackHandle
MpackHandle.AddExt(reflect.TypeOf(time.Time{}), 1, MpackHandle.TimeEncodeExt, MpackHandle.TimeDecodeExt)

I tried this which doesn't work (encodeTime is undefined):

timeEncExt := func(rv reflect.Value) ([]byte, error) {
    return encodeTime(rv.Interface().(time.Time)), nil
}
timeDecExt := func(rv reflect.Value, bs []byte) error {
    tt, err := decodeTime(bs)
    if err == nil {
        rv.Set(reflect.ValueOf(tt))
    }
    return err
}

MpackHandle.AddExt(reflect.TypeOf(time.Time{}), 1, timeEncExt, timeDecExt)

The below snippet produces perhaps sub-optimal msgpack.

Notice that the struct is created from small numbers (even if big types), in serialization with MsgPack we should be able to create these numbers without any standalone type descriptor by using the positive FixInt type from the spec:

positive fixint 0xxxxxxx    0x00 - 0x7f

If you look at the output of the snippet you'll notice several 0xcc bytes in there describing each of the integers as a uint8 when those bytes don't need to exist.

Removing the 0xcc bytes and decoding with this package produces the expected result, so it seems to be an encoding problem only.

package main

import (
    "github.com/ugorji/go/codec"
    "bytes"
    "log"
)

type bstruct struct {
    A, B, C uint64
}

func main() {
    b := bstruct{1, 2, 3}

    log.Println(b)

    var m codec.MsgpackHandle
    m.StructToArray = true
    m.WriteExt = true

    var buf bytes.Buffer
    encoder := codec.NewEncoder(&buf, &m)
    if err := encoder.Encode(b); err != nil {
        log.Fatal("Failed to encode:", err)
    }

    log.Printf("%#v", buf.Bytes())

}

Output:

2013/11/14 17:24:29 {1 2 3}
2013/11/14 17:24:29 []byte{0x93, 0xcc, 0x1, 0xcc, 0x2, 0xcc, 0x3}

I'm sorry I couldn't get a smaller test case for this, I'm short on time this morning. This is trying to decode a payload from Python (hence the ugly string constant).

If you name the Bstruct's *Astruct field instead of embedding it, it decodes as expected. If you don't you get the error message:

2013/11/15 09:01:54 Failed to decode to struct:
codec.decoder: readContainerLen: Unrecognized descriptor byte: hex: 95, dec: 149

package main

import (
    "github.com/ugorji/go/codec"
    "bytes"
    "log"
    "github.com/davecgh/go-spew/spew"
)

type Astruct struct {
    Abyte1 []byte
    Abyte2 []byte
    Auint1 uint64
    Auint2 uint64
    Auint3 uint8
}

type Bstruct struct {
    *Astruct
    Bint1  int32
    Buint1 uint64
    Buint2 uint64
    Bint2  int32
    Bint3  int32
}

type Cstruct struct {
}

type Dstruct struct {
    Dmap map[string]*Bstruct
    Dslice []*Cstruct
}

var test = []byte("\x92\x81\xa4test\x96\x95\xda\x00 \x13\n\x1e\x12" +
            "\x15t\xf6y\x8e\xf6\xcd\xc3\xde)t2\xfc\xb7\xe0\xcf\xbd\xd9" +
            "\xbd\xc2$\xdc\x01\xc5=89\xc9\xda\x00 `\xcb1\xbe+\x1aGr" +
            "\x1a\x06\xef\xf3)\xc4\xcf\xef\xfcp\xac\xa9-\n\xe1TV\xb0" +
            "\xf8\xd5EX\xceH$\x0b\x01\x0b\x00\x00\x00\x00\x90")

func main() {
    var m codec.MsgpackHandle

    d := Dstruct{}
    decoder := codec.NewDecoder(bytes.NewBuffer(test), &m)
    if err := decoder.Decode(&d); err != nil {
        log.Println("Failed to decode to struct:", err)
    } else {
        log.Println(d)
    }

    var i interface{}
    decoder = codec.NewDecoder(bytes.NewBuffer(test), &m)
    if err := decoder.Decode(&i); err != nil {
        log.Println("Failed to decode to interface:", err)
    } else {
        spew.Dump(i)
    }
}

Output:

2013/11/15 09:01:54 Failed to decode to struct: codec.decoder: readContainerLen: Unrecognized descriptor byte: hex: 95, dec: 149
([]interface {}) {
 (map[interface {}]interface {}) {
  (string) "test": ([]interface {}) {
   ([]interface {}) {
    ([]uint8) {
     00000000  13 0a 1e 12 15 74 f6 79  8e f6 cd c3 de 29 74 32  |.....t.y.....)t2|
     00000010  fc b7 e0 cf bd d9 bd c2  24 dc 01 c5 3d 38 39 c9  |........$...=89.|
    },
    ([]uint8) {
     00000000  60 cb 31 be 2b 1a 47 72  1a 06 ef f3 29 c4 cf ef  |`.1.+.Gr....)...|
     00000010  fc 70 ac a9 2d 0a e1 54  56 b0 f8 d5 45 58 ce 48  |.p..-..TV...EX.H|
    },
    (int64) 36,
    (int64) 11,
    (int64) 1
   },
   (int64) 11,
   (int64) 0,
   (int64) 0,
   (int64) 0,
   (int64) 0
  }
 },
 ([]interface {}) {
 }
}

If I define a map with value type of interface{} (I'm sending data over the wire in data structures including strings and timestamps) then the map is de-serialised as a byte array:

2013/08/06 02:53:33 Buffer is  [130 169 115 111 109 101 102 105 101 108 100 170 115 111 109 101 115 116 114 105 110 103 167 116 104 101 116 105 109 101 171 239 82 0 87 29 47 122 62 127 0 60]
2013/08/06 02:53:33 Decoded is map[somefield:[115 111 109 101 115 116 114 105 110 103] thetime:[239 82 0 87 29 47 122 62 127 0 60]]

The program to generate this is:

package main

import (
    "github.com/ugorji/go/codec"
    "time"
    "reflect"
    "log"
)

type timeContainer map[string]interface{}

var (
    mh codec.MsgpackHandle
    timeTyp = reflect.TypeOf(time.Time{})
    source = timeContainer {
        "somefield": "somestring",
        "thetime": time.Now(),
    }
)

func main() {
    mh.AddExt(timeTyp, 1, mh.TimeEncodeExt, mh.TimeDecodeExt)

    // Encode the value

    var encode_out []byte
    enc := codec.NewEncoderBytes(&encode_out, &mh)
    encode_err := enc.Encode(&source)
    if encode_err != nil {
        log.Fatal("Ooops, error: ", encode_err)
    }

    log.Println("Buffer is ", encode_out)


    // Decode the value

    var b timeContainer
    dec := codec.NewDecoderBytes(encode_out, &mh)
    decode_err := dec.Decode(&b)
    if decode_err != nil {
        log.Fatal("Ooops, error: ", decode_err)
    }

    log.Printf("Decoded is %v", b)
}

Any pointers towards getting values defined as interface{} decoded properly would be great.

See https://gist.github.com/methane/bffc908568deb96c2a27

Hello,

I'm trying to use your MessagePack implementation to encode an decode a map[string]interface{} but the behavior I'm getting isn't intuitive and I can't find any doc about it.

Basically here's what happen:

• I encode map[string]interface{}{"hello":"world"} with the MsgpackHandle codec
• then decode with the same MsgpackHandle codec but I get map[hello:[119 111 114 108 100]] in return

It looks like the string was encoded or decoded as a slice of bytes or something similar.

Ideally I would like the map value to be encoded and decoded as a UTF-8 string, is there something special I'm supposed to do to get the Encoder/Decoder to behave this way?

I'm new to go so maybe I'm not understanding the type system very well though, sorry if that problem isn't related to the MessagePack implementation.

Thanks for your time.

since this library support map[string]interface{} type, I tried to use it for pacakge struct, what I find is that decode/encode data dismatch, my test code:

func TestCodec(t *testing.T) {
    var (
        mh  codec.MsgpackHandle
        rw  bytes.Buffer
    )
    mh.MapType = reflect.TypeOf(map[string]interface{}(nil))
    enc := codec.NewEncoder(&rw, &mh)

    item := make(map[string]interface{})
    item["key"] = int32(123)
    item["key2"] = "string"

    err := enc.Encode(&item)
    if err == nil {
        t.Log("Encode OK", "type", reflect.TypeOf(item["key"]), reflect.TypeOf(item["key2"]))
    }

    var item_decoded map[string]interface{}

    dec := codec.NewDecoder(&rw, &mh)
    err = dec.Decode(&item_decoded)
    if err == nil {
        t.Log("Decode OK", "type", reflect.TypeOf(item_decoded["key"]), reflect.TypeOf(item_decoded["key2"]))
    }
}

the result:

proto_test.go:63: Encode OK type int32 string
proto_test.go:71: Decode OK type int64 []uint8

update:

I change the handle config mh.RawToString = true, now the int type still not match:

proto_test.go:64: Encode OK type int32 string
proto_test.go:72: Decode OK type int64 string

msgpack spec include int8 int16 int32 many int families, this result make me confuse.

Hi there--thanks for providing this library!

I'm in the process of upgrading from your now-deprecated go-msgpack library, where I was using the convenience methods Marshal and Unmarshal. Using this new library, Encode works as a perfect drop in replacement for Marshal, but when I try to decode I get the following runtime error:
"interface is []uint8, not string"
where casting to a string (as it was encoding as a string) worked before. Is this a bug, or has the usage changed?

My Marshal/Unmarshal replacement are:

var mh codec.MsgpackHandle

//Convenience methods for simple encoding/decoding
func MsgpackEncode(val interface{}) (packed []byte, err error) {
enc := codec.NewEncoderBytes(&packed, &mh)
err = enc.Encode(val)
return
}

func MsgpackDecode(packed []byte, val interface{}) (err error) {
dec := codec.NewDecoderBytes(packed, &mh)
err = dec.Decode(&val)
return

}

Appreciate the help.

When running codecgen against the ffjson test set I got this error:

c:\GoPath\src\github.com\pquerna\ffjson\tests>codecgen -o ff_codec.go ff.go
codecgen error: Error running go run codecgen-1424087427033171900.generated.go.
Error: exit status 1. stdout/err: panic: 27217:22: expected '}', found 'IDENT' i
nt (and 10 more errors)

goroutine 1 [running]:
main.main()
        c:/GoPath/src/github.com/pquerna/ffjson/tests/codecgen-14240874270331719
00.generated.go:422 +0x1103f
exit status 2

go get -u github.com/pquerna/ffjson
cd github.com/pquerna/ffjson/tests  (in gopath)
codecgen -o ff_codec.go ff.go

Also, you might want to update the information on your blog - all of the points you mention about ffjson is no longer true. Decoding is now supported, struct renaming works, and generating values.go:TestStruct works.

The below snippet fails to decode with the error:

2013/11/15 09:16:22 Could not decode a: codec.decoder: readContainerLen: 
Unrecognized descriptor byte: hex: 90, dec: 144

Replacing the []byte field with with an array of pointers to structs, or even []int makes it decode without problems.

package main

import (
    "github.com/ugorji/go/codec"
    "log"
    "bytes"
)

var test1 = []byte{0x91, 0x90}

type Astruct struct {
    Anarray []byte
}

func main() {
    var mph codec.MsgpackHandle

    var a = Astruct{}

    da := codec.NewDecoder(bytes.NewBuffer(test1), &mph)
    if err := da.Decode(&a); err != nil {
        log.Fatalln("Could not decode a:", err)
    }
    log.Println(a)
}

To create a msgpack-rpc message (http://wiki.msgpack.org/display/MSGPACK/RPC+specification) the message ID must be specified as a uint32, which seems impossible from reading the source code of the current implementation. The encoder chooses by fitting the passed uint64 in the smallest msgpack container.

Is it possible to have the encoder detect the source type and keep the size? Maybe an option in the handler to compress into smaller types could be added?

My request struct looks like this:

type Request struct {
   id     uint32
   Method string
   Params []interface{}
}

Packing is done with this method:

func (r *Request) Packed() []byte {
    a := make([]interface{}, 0)

    a = append(a, 0)
    a = append(a, r.id)
    a = append(a, r.Method)
    a = append(a, r.Params)

    var b []byte
    var mh codec.MsgpackHandle

    enc := codec.NewEncoderBytes(&b, &mh)
    err := enc.Encode(a)
    if err != nil {

    }

    return b
}

Unpacking is done as follows:

func UnpackRequest(b []byte) *Request {
    var a []interface{}
    mh := codec.MsgpackHandle{RawToString: true}

    dec := codec.NewDecoderBytes(b, &mh)
    err := dec.Decode(&a)
    if err != nil {

    }

    r := &Request{
        id:     uint32(a[1].(uint64)), // Ugly conversion has to be done here.
        // Prefered way:
        // id: a[1].(uint32),
        Method: a[2].(string),
        Params: a[3].([]interface{}),
    }

    return r
}

Please, document is it safe to declare a global MsgpackHandle and/or Encoder/Decoder objects and use them concurrently from different goroutines.

This is more of a question/discussion than an issue.

MsgPack supports FixStr, as well as Str8, Str16, Str32 and also Bin8, Bin16, Bin32.

When encoding the sample from issue #22 without the bug inducing bit (specifically: the type declaration of binary).. I noticed that the byte slice is being encoded as FixStr for small lengths, and StrX (X being the accompanying size) instead of the BinX variants.

I think maybe this could be a potential issue between different clients who are expecting a specific schema.

Thoughts on this?

Needs to separate map k:v pairs with commas - ',' - use case below w/o codecgen output. To complete "go test" case run "codecgen -o codecdemo_codec.go codecdemo.go" first.

codecdemo.go:

package codecdemo

import (
"github.com/ugorji/go/codec"
"strconv"
"time"
)

type MapVal struct {
Name string
Time time.Time
}

type Entry struct {
Vals map[string]*MapVal
}

type Wrapper struct {
Entries []*Entry
}

func Codecdemo() ([]byte, error) {
w := new(Wrapper)
w.Entries = make([]_Entry, 1)
e := new(Entry)
w.Entries[0] = e
w.Entries[0].Vals = make(map[string]_MapVal, 0)

for i := 0; i < 4; i++ {
    m := new(MapVal)
    m.Name = strconv.Itoa(i + 1)
    m.Time = time.Now()
    w.Entries[0].Vals[m.Name] = m
}

b := make([]byte, 0)
h := new(codec.JsonHandle)
enc := codec.NewEncoderBytes(&b, h)
if err := enc.Encode(w); err != nil {
    return nil, err
}

return b, nil

}

codecdemo_test.go:

import (
"encoding/json"
"testing"
)

func TestCodecDemo(t *testing.T) {
val, err := Codecdemo()
if err != nil {
t.Fatal(err)
}

m := new(Wrapper)
if err = json.Unmarshal(val, m); err != nil {
    println("val:", string(val))
    t.Fatal(err)
}

println("val:", string(val))

}

hi, I have a simple struct pass from cpp to golang, struct such as
struct { std::string A; int B };
here's problem:
msgpack cpp lib will encode such struct to array, but golang lib will decode map to struct automatically, not array.
here's what I want:
I hope golang lib will decode such array to golang struct, just omit field name.
is there any way to do such thing?
or what's your suggestion?

This records the issue found by @mattrichards in PR #54 .

errors could occur although some bytes may have been successfully read.

When encoding a structure to an array, an omitted field will return an error:

Error: reflect: call of reflect.Value.Type on zero Value

package main

import (
    "bytes"
    "fmt"

    "github.com/ugorji/go/codec"
)

type SrcIdDate struct {
    Sid          string
    SDate, EDate string
    Option       string `codec:",omitempty"`
}

func main() {
    var (
        mh  codec.MsgpackHandle
        buf bytes.Buffer
    )

    s1 := SrcIdDate{Sid: "ABC", SDate: "20000101", EDate: "20101031"}
    mh.EncodeOptions.StructToArray = true
    enc := codec.NewEncoder(&buf, &mh)
    err := enc.Encode(s1)
    if err != nil {
        fmt.Printf("Error: %v\n", err)
    } else {
        fmt.Printf("Success: %v\n", buf.Bytes())
    }
}

This doesn't occur when encoding as a map or if the field is not empty.

I can see some confusion if the optional field wasn't at the end, but I am trying to process some previously generated data.

I need to do a "canonical" encoding in CBOR -- much as described in http://tools.ietf.org/html/rfc7049#section-3.9. In other words, I need to make some calls to the encoder that when given the same inputs, have identical binary outputs. (Though it's probably not relevant, in the bigger picture, this is for feeding into a cryptographic hash, so yes, the input->binaryoutput mapping has to be identical every time.)

How can I do this?

I think codec already mostly does the right thing with ints and so forth, but maps are a sticking point. Here's what I considered so far:

• It looks like the default path accepts the random key ordering of golang map, so that's out.
• cborEncDriver has enough exported methods on it that I might be able to get the trick done by hand (and that'd be fine; since this is finicky work with byte layouts, it's fair if my code has to do fairly manual byte concatenations, as long as it's not totally re-copying everything down to the magic constants and int/type layouts)... except it (and a bunch of other important magic constants) are not exported, so I can't just initialize it.
• fast-path.go.tmpl seemed like it offered a glimmer of hope for a second, because it's doing the things I need to do... but it looks like that code only works from within the codec package (lo and behold it's using those unexported encoder drivers).
• ... Ah ha! I can use GenHelperEncoder to get access to all the fun stuff. It's documented as not for use, but...

Option 4 is the only thing I've found that's capable of doing what's needed to output a consistent/canonical form. Is there anything I've missed that's a better way of addressing this? Are there any code changes I could help PR that would make this better?

At f8f25d7 with go1.2rc3 (amd64/linux), I can encode a slice of interface R pointers. But I cannot decode them. Is there a trick to this? Example:

package main

import (
    "bytes"
    "fmt"

    "github.com/ugorji/go/codec"
)

type R interface {
    String() string
}

type Block struct {
    Name string
    Rs   []*R
}

type Registration struct {
    Reg string
}

func (r Registration) String() string {
    return fmt.Sprintf("Registration: Reg: '%s'", r.Reg)
}

func NewBlock(name string) *Block {
    blk := &Block{
        Name: name,
        Rs:   make([]*R, 0),
    }
    return blk
}

func main() {
    name := "hello"
    blk := NewBlock(name)

    var r1, r2 R
    r1 = Registration{Reg: "and-a-one"}
    r2 = Registration{Reg: "and-a-two"}

    blk.Rs = append(blk.Rs, &r1)
    blk.Rs = append(blk.Rs, &r2)
    bincEncodeDecode(blk)
}

func bincEncodeDecode(blk *Block) {

    var bh codec.BincHandle
    b := make([]byte, 0)
    //var b []byte

    enc := codec.NewEncoderBytes(&b, &bh)
    err := enc.Encode(blk)
    if err != nil {
        panic(err)
    }
    fmt.Printf("encoded blk in bytes is: %v\n", b)

    fmt.Printf("pre-encoding blk is: %v\n", blk)
    fmt.Printf("pre-encoding blk.name is: '%v'\n", blk.Name)

    //var blk2 interface{}
    blk2 := NewBlock("")

    dec := codec.NewDecoder(bytes.NewBuffer(b), &bh)
    err = dec.Decode(blk2)

    if err != nil {
        panic(err) // panic: codec.decoder: decodeValue: Cannot decode non-nil codec value into nil main.R (1 methods)
    }

    fmt.Printf("decoded blk2 is: %v\n", blk2)
    fmt.Printf("blk2.name is: '%v'\n", blk2.Name)

    /* output:
    go run bincdecode.go
    encoded blk in bytes is: [118 180 1 4 78 97 109 101 73 104 101 108 108 111 180 2 2 82 115 102 117 180 3 3 82 101 103 77 97 110 100 45 97 45 111 110 101 117 176 3 77 97 110 100 45 97 45 116 119 111]
    pre-encoding blk is: &{hello [0xc21000aa20 0xc21000aa30]}
    pre-encoding blk.name is: 'hello'
    panic: codec.decoder: decodeValue: Cannot decode non-nil codec value into nil main.R (1 methods)

    goroutine 1 [running]:
    runtime.panic(0x669080, 0xc21000ab90)
        /usr/local/go1.2rc3/go/src/pkg/runtime/panic.c:266 +0xb6
    main.bincEncodeDecode(0xc210066810)
        /home/me/dev/bincdecode.go:71 +0x4f2
    main.main()
        /home/me/dev/bincdecode.go:45 +0x327
    exit status 2

    Compilation exited abnormally with code 1 at Mon Dec 16 20:54:28
    */
}

package main

import (
    "github.com/ugorji/go/codec"
    "bytes"
    "log"
)

type A struct {
    Aint int
}

type B struct {
    *A
    Bint int
}

type C struct {
    *A
    Cint int
}

type D struct {
    Structs map[string]*B
    Array []*C
}

func main() {
    obj := &D{
        Structs: map[string]*B{
            "test": &B{&A{5}, 6},
        },
    }

    var m codec.MsgpackHandle
    m.StructToArray = true

    var buf bytes.Buffer
    encoder := codec.NewEncoder(&buf, &m)
    if err := encoder.Encode(obj); err != nil {
        log.Fatal("Failed to encode:", err)
    }

    decoder := codec.NewDecoder(&buf, &m)
    d := D{}
    if err := decoder.Decode(&d); err != nil {
        log.Fatal("Failed to decode:", err)
    }

    log.Println(d)
}

This code receives the error:
reflect: call of reflect.Value.Field on ptr Value

It runs correctly when the struct pointers are named and not embedded OR the StructToArray option is turned off.

Is this expected to fail?

package main

import (
    "bytes"
    "github.com/ugorji/go/codec"
)

var msgpackHandle = new(codec.MsgpackHandle)

type A struct {
    anInt int
}

type B struct {
    *A
    moreInt int
}

func main() {
    var buf bytes.Buffer
    encoder := codec.NewEncoder(&buf, msgpackHandle)
    err := encoder.Encode(&B{&A{5}, 6})
    if err != nil {
        panic(err)
    }
}

Please provide a mechanism to do Steaming RPC as an extension.
I know net/rpc lacks that feature currently.

Notice the type definition of binary to a []byte. This causes the []byte to not be serialized as a regular []uint8, but as a needlessly verbose array of bytes.

package main

import (
    "github.com/ugorji/go/codec"
    "bytes"
    "log"
)

type binary []byte

type bstruct struct {
    B binary
}

func main() {
    b := bstruct{
        []byte{5, 7, 8},
    }

    log.Println(b)

    var m codec.MsgpackHandle
    m.StructToArray = true

    var buf bytes.Buffer
    encoder := codec.NewEncoder(&buf, &m)
    if err := encoder.Encode(b); err != nil {
        log.Fatal("Failed to encode:", err)
    }

    log.Printf("%#v", buf.Bytes())

    decoder := codec.NewDecoder(&buf, &m)
    b = bstruct{}
    if err := decoder.Decode(&b); err != nil {
        log.Fatal("Failed to decode:", err)
    }

    log.Println(b)
}

The output is as follows:

2013/11/14 09:05:01 {[5 7 8]}
2013/11/14 09:05:01 []byte{0x91, 0x93, 0xcc, 0x5, 0xcc, 0x7, 0xcc, 0x8}
2013/11/14 09:05:01 {[5 7 8]}

Notice the 0xcc in there, describing the single byte each time for all bytes in the array.

Could it support something like this:

rpcServer := rpc.NewServer()
rpcServer.Register(mything)
rpcServer.ServeCodec(rpcCodec)
rpcServer.Accept(listener)

I can't see a way to get the rpcCodec with already accepting a connection though.

Hi,

I has this snippet of code:

type Encoder interface {
    Encode(v interface{}) error
}

type Decoder interface {
    Decode(v interface{}) error
}

type (
    EncoderFactory func(w io.Writer) Encoder
    DecoderFactory func(r io.Reader) Decoder
)

type Codec struct {
    EncodingName string
    NewEncoder   EncoderFactory
    NewDecoder   DecoderFactory
}

var JsonCodec = Codec{
    "json",
    func(w io.Writer) Encoder {
        return json.NewEncoder(w)
    },
    func(r io.Reader) Decoder {
        return json.NewDecoder(r)
    },
}

var MsgpackCodec = Codec{
    "msgpack",
    func(w io.Writer) Encoder {
        var mh codec.MsgpackHandle
        return codec.NewEncoder(w, &mh)
    },
    func(r io.Reader) Decoder {
        var mh codec.MsgpackHandle
        return codec.NewDecoder(r, &mh)
    },
}

so that I can use either json or msgpack interchangeably. Now to my surprise, when I pass interface{} or map[string]interface{} to Decode, I get different results from encoding/json Decoder and your codec Decoder, namely the former one returns values as strings while the latter one as []uint8. Is that desired? Is it for some kind of performance reasons? I think that it would be nice to be somehow compatible...

Thanks!

Regards,
Ondrej Kupka

how can i use msgpack in golang?