From [email protected] on November 08, 2014 19:44:30

What steps will reproduce the problem? 1. send a big protocol buffer
2. on receipt, try to decode it
3. watch it crash the program trying to decode a varint

Here is the stack trace:

unexpected fault address 0x7f8581ad3175
fatal error: fault
[signal 0xb code=0x1 addr=0x7f8581ad3175 pc=0x632e7f]

goroutine 38 [running]:
runtime.throw(0x145af62)
/home/jaten/pkg/go1.3.1/go/src/pkg/runtime/panic.c:520 +0x69 fp=0x7f85dc27d178 sp=0x7f85dc27d160
runtime.sigpanic()
/home/jaten/pkg/go1.3.1/go/src/pkg/runtime/os_linux.c:240 +0x13f fp=0x7f85dc27d190 sp=0x7f85dc27d178
code.google.com/p/gogoprotobuf/proto.(_Buffer).DecodeVarint(0xc22f541ba0, 0x0, 0x0, 0x0)
/home/jaten/go/src/code.google.com/p/gogoprotobuf/proto/decode.go:92 +0x8f fp=0x7f85dc27d1b0 sp=0x7f85dc27d190
code.google.com/p/gogoprotobuf/proto.(_Buffer).DecodeRawBytes(0xc22f541ba0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
/home/jaten/go/src/code.google.com/p/gogoprotobuf/proto/decode.go:179 +0x73 fp=0x7f85dc27d280 sp=0x7f85dc27d1b0
code.google.com/p/gogoprotobuf/proto.(_Buffer).dec_slice_ref_struct(0xc22f541ba0, 0xc2081007e0, 0x0, 0xc3779eeb00, 0x0, 0x0)
/home/jaten/go/src/code.google.com/p/gogoprotobuf/proto/decode_gogo.go:112 +0xe2 fp=0x7f85dc27d328 sp=0x7f85dc27d280
code.google.com/p/gogoprotobuf/proto.(_Buffer).dec_slice_ref_struct_message(0xc22f541ba0, 0xc2081007e0, 0xc3779eeb00, 0x0, 0x0)
/home/jaten/go/src/code.google.com/p/gogoprotobuf/proto/decode_gogo.go:137 +0x52 fp=0x7f85dc27d360 sp=0x7f85dc27d328
code.google.com/p/gogoprotobuf/proto.(_Buffer).unmarshalType(0xc22f541ba0, 0x7f85dfe8c1d8, 0xa3d080, 0xc28d2ba3f0, 0xc3779eeb00, 0xc3779eeb00, 0x0, 0x0)
/home/jaten/go/src/code.google.com/p/gogoprotobuf/proto/decode.go:404 +0xd2f fp=0x7f85dc27d6b8 sp=0x7f85dc27d360
code.google.com/p/gogoprotobuf/proto.(_Buffer).dec_struct_message(0xc22f541ba0, 0xc2081006c0, 0xc22136d9e0, 0x0, 0x0)
/home/jaten/go/src/code.google.com/p/gogoprotobuf/proto/decode.go:676 +0x2d7 fp=0x7f85dc27d790 sp=0x7f85dc27d6b8
code.google.com/p/gogoprotobuf/proto.(_Buffer).unmarshalType(0xc22f541ba0, 0x7f85dfe8c1d8, 0xa69780, 0xc28d2ba310, 0xc22136d900, 0xc22136d9e0, 0x0, 0x0)
/home/jaten/go/src/code.google.com/p/gogoprotobuf/proto/decode.go:404 +0xd2f fp=0x7f85dc27dae8 sp=0x7f85dc27d790
code.google.com/p/gogoprotobuf/proto.(_Buffer).Unmarshal(0xc22f541ba0, 0x7f85bd02e3c0, 0xc22136d9e0, 0x0, 0x0)
/home/jaten/go/src/code.google.com/p/gogoprotobuf/proto/decode.go:334 +0x234 fp=0x7f85dc27dbb0 sp=0x7f85dc27dae8
code.google.com/p/gogoprotobuf/proto.UnmarshalMerge(0x7f8581a60030, 0xe59f46d, 0xe59f46d, 0x7f85bd02e3c0, 0xc22136d9e0, 0x0, 0x0)
/home/jaten/go/src/code.google.com/p/gogoprotobuf/proto/decode.go:314 +0x134 fp=0x7f85dc27dc20 sp=0x7f85dc27dbb0
code.google.com/p/gogoprotobuf/proto.Unmarshal(0x7f8581a60030, 0xe59f46d, 0xe59f46d, 0x7f85bd02e3c0, 0xc22136d9e0, 0x0, 0x0)
/home/jaten/go/src/code.google.com/p/gogoprotobuf/proto/decode.go:300 +0x7a fp=0x7f85dc27dc60 sp=0x7f85dc27dc20
[call from my code into gogoprotobuf Unmarshal() occurs here] What version of the product are you using? On what operating system? ubuntu 12.04 /amd64 / linux 3.2.0-69-generic #103-Ubuntu SMP Tue Sep 2 05:02:14 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Please provide any additional information below. I'm sending 114MB in a protocol buffer object when this happens, it doesn't seem to happen on very small buffers.

Original issue: http://code.google.com/p/gogoprotobuf/issues/detail?id=29

The default implementation requires a flag --go_out=plugins=grpc to generate services (see https://github.com/grpc/grpc-common/tree/master/go#optional---rebuilding-the-generated-code), but gogoproto always generates them, which is sometimes undesired (e.g. when a project is generating custom code for rpc that isn't grpc)

From awalterschulze on October 08, 2013 17:29:16

Write tests with gogoprototest and generated nullable, marshalling, unmarshalling and size code to create deterministic behaviour.

Original issue: http://code.google.com/p/gogoprotobuf/issues/detail?id=1

From [email protected] on November 15, 2014 02:53:00

If you read the code, you'll see varintReader.maxSize is never used.

Original issue: http://code.google.com/p/gogoprotobuf/issues/detail?id=32

From [email protected] on August 15, 2014 07:33:17

What steps will reproduce the problem? 1. Use 'import_prefix=protobuf/common' What is the expected output? What do you see instead? Expected:
import proto "code.google.com/p/gogoprotobuf/proto"
import "protobuf/common/a.pb"

What I see:
import proto "protobuf/common/code.google.com/p/gogoprotobuf/proto"
import "protobuf/common" What version of the product are you using? On what operating system? OSX Please provide any additional information below.

Original issue: http://code.google.com/p/gogoprotobuf/issues/detail?id=24

This issue was originally reported here: #39.

I see that when you have a field named String golang/protobuf generates that fieldname as String_, since this field conflict with the generated String method.

I think this would be a great solution to the Size method and fieldname conflict problem.

From [email protected] on January 10, 2014 02:12:11

Using the included source, it seems the import line is renamed to "proto1", but this new name is not used in the generated Unmarshal code.

/*

protoc -I$GOPATH/src -I. --gogo_out=. two.proto

$ go build two.pb.go

command-line-arguments

./two.pb.go:60: undefined: proto

$ grep ^package two.pb.go
package proto

$ grep '^import proto1' two.pb.go
import proto1 "code.google.com/p/gogoprotobuf/proto"

*/
package proto;

import "code.google.com/p/gogoprotobuf/gogoproto/gogo.proto";

option (gogoproto.unmarshaler_all) = true;

message Foo {
required uint64 bar = 1;
}

Original issue: http://code.google.com/p/gogoprotobuf/issues/detail?id=8

This is due to lines 523 and 525 of plugin/unmarshal/unmarshal.go (in rev d59ce9e). If the actual value of the field on the wire was an empty byte string, then this will result in the deserialized field having a value of nil rather than []byte{}.

Will send a pull request shortly

main.go:

package main

//go:generate protoc -I. -I$GOPATH/src -I$GOPATH/src/github.com/gogo/protobuf/protobuf --gogo_out=. repro.proto

type T uint32

func main() {
}

repro.proto:

package repro;

option go_package = "main";

import "github.com/gogo/protobuf/gogoproto/gogo.proto";

option (gogoproto.unmarshaler_all) = true;

message M {
  required uint32 X = 1 [(gogoproto.nullable) = false, (gogoproto.customtype) = "T"];
}

$ go generate && go build
# _/home/tv/src/2014/protobuf-bug-type
./repro.pb.go:69: invalid operation: m.X |= uint32(b) & 127 << shift (mismatched types T and uint32)

From [email protected] on August 21, 2014 22:02:47

The Marshal() and Unmarshal() methods generated by gogoprotobuf ignore the "required" specification. Although using "required" is generally discouraged, it can make sense in some circumstances -- and moreover, silently ignoring "required" (without prominent documentation) seems like not-great behavior.

Possible remedies:

1. Providing an option to have Marshal() and Unmarshal() obey the "required" directive.
2. Perhaps enabling #1 by default.
3. Prominently documenting this behavior, especially if #1 is not the default behavior.

Original issue: http://code.google.com/p/gogoprotobuf/issues/detail?id=26

This idea was first mentioned in #41
, but I thought I'd rather split it out into a separate issue.

If you would like the generated code to rather import github.com/golang/protobuf/proto instead of github.com/gogo/protobuf/proto please vote here, by commenting below.

My plan is to add another extension so that you can set certain files to rather import golang/protobuf.
This way it is backwards compatible with the current implementation.

I also suspect that these users will be users that don't want to use extensions or import gogo.proto in their proto files.
The solution to this problem will be provided by the solution to #39

In other words with a protoc-gen-gogovanity binary you could get all the code generation you want without depending on gogoprotobuf.

There are a bunch of dropbox internal customization which I'm interested in upstreaming. In particular:

1. We've rename Size() to ProtoSize()
2. We've enable marshaller, unmarshaller, and sizer by default

To avoid breaking the rest of the world, I was thinking of creating a struct which would holds the default values (and we'll only patch this struct within dropbox). something like

type DefaultParams struct {
   sizeMethodName string
   isMarshaller bool
   isUnmarshaller bool
   isSizer bool
}

the world would use:

params := DefaultParams{"Size", false, false, false}

while dropbox would override the defaults to:

params := DefaultParams{"ProtoSize", true, true, true}

Is this acceptable?

Also, on the experimental side, I want to backport plugins from https://github.com/dropbox/goprotoc to generate proper mutator APIs (we've got burn by goproto's lack of proper mutator api a few of time recently, especially around repeated field). Is this something you would be interested in?

Thanks,
Patrick

From [email protected] on May 29, 2014 18:14:22

Hey.

What steps will reproduce the problem?

1. create a proto file like

package mypackage;
import "code.google.com/p/gogoprotobuf/gogoproto/gogo.proto";
option (gogoproto.marshaler_all) = true;

message test {}

1. save it with name like my-package.proto (note the dash in the name)
2. run protoc --gogo_out=. -I$GOROOT/src -I. my-package.proto

What is the expected output? What do you see instead?

You'll get an error like:

2014/05/29 20:03:07 protoc-gen-gogo: error:bad Go source code was generated: 71:21: expected '(', found '-' (and 2 more errors) What version of the product are you using? On what operating system? - latest gogoprotobuf master branch (fb9da01)

• go version go1.2 linux/amd64
• Linux antoxa-suse.site 3.7.10-1.16-desktop #1 SMP PREEMPT Fri May 31 20:21:23 UTC 2013 (97c14ba) x86_64 x86_64 x86_64 GNU/Linux Please provide any additional information below. i've narrowed it down to
  func (p *marshalto) Generate(file *generator.FileDescriptor) {
  ...
  p.localName = generator.FileName(file)

which transforms filename into camel cased name to use for function names.
camel casing handles underscores fine, but not dashes, and you get function names like "encodeVarintMy-package".

Naive patch - attached.

Attachment: dashes-in-filenames.patch

Original issue: http://code.google.com/p/gogoprotobuf/issues/detail?id=16

I was playing with customtypes and realized my type already implements http://golang.org/pkg/encoding/#BinaryMarshaler etc, but this wasn't what gogoprotobuf wants -- and then I started wondering, why (apart from chronological order of things showing up)?

From [email protected] on January 10, 2014 03:06:49

A field named "size" conflicts with the Size method added by the sizer plugin.

option (gogoproto.sizer_all) = true;
option (gogoproto.marshaler_all) = true;

message Foo {
required uint64 size = 1 [(gogoproto.nullable) = false];
}

foo.pb.go:NN: type Foo has both field and method named Size
foo.pb.go:NN: cannot call non-function m.Size (type uint64)

Any idea how to proceed? I would like to keep the details of Go out of the protobuf -- imagine a situation where the protobuf is for compatibility with e.g. Google APIs.

Size() used by Marshal() internally could be renamed to size(), but I do want to length-prefix my messages, so I'll need to call the Size method from outside too.

Original issue: http://code.google.com/p/gogoprotobuf/issues/detail?id=9

From [email protected] on August 09, 2014 16:48:46

What steps will reproduce the problem? 1. Have a vendored raft such as: https://github.com/influxdb/influxdb/blob/master/_vendor/raft/protobuf/append_entries_request.pb.go#L113 2. Try to build project
3. Fail on _vendor/raft/protobuf/append_entries_request.pb.go:113: undefined: proto.ErrWrongType What is the expected output? What do you see instead? Expected output is that it works. Instead, lots of undefined messages

Original issue: http://code.google.com/p/gogoprotobuf/issues/detail?id=23

From awalterschulze on June 12, 2014 14:00:23

see TestBugUuid, with marshaler_all = true in thetest.proto

Original issue: http://code.google.com/p/gogoprotobuf/issues/detail?id=17

From awalterschulze on August 07, 2014 13:59:23

protoc-gen-gogo/generator/helper.go

Add this method

func (d _FileDescriptor) AllMessages() []_Descriptor {
msgs := d.desc
for _, i := range d.imp {
obj, ok := i.o.(*Descriptor)
if ok {
msgs = append(msgs, obj)
}
}
return msgs
}

In each plugin there are gogoproto.Has/Is functions
At the moment they reference the current file, but they should reference the file in which the message/field was originally defined.

If the message was publicly imported its code generation should depend upon the file in which it was defined not the file which publicly imports it.

Most of it should work after this work is done.

Original issue: http://code.google.com/p/gogoprotobuf/issues/detail?id=22

From awalterschulze on October 08, 2013 17:30:09

fix proto.Size for nullable=false and embedded fields when code was not generated.

Original issue: http://code.google.com/p/gogoprotobuf/issues/detail?id=2

This is pretty stable on the proto3 branch.

From [email protected] on June 23, 2014 19:59:04

If the incoming data is corrupt or malicious,
the call to http://golang.org/pkg/encoding/binary/#Uvarint in http://godoc.org/code.google.com/p/gogoprotobuf/io#NewDelimitedReader ReadMsg can cause a slice index out of bounds panic.

The code in https://code.google.com/p/gogoprotobuf/source/browse/io/varint.go#98 should check for lenLen <= 0 and handle it somehow (0 is internal error, <0 is corrupt input). I don't think there's any graceful way to recover, so maybe the cleanest way is to have the error become sticky; all further ReadMsg calls should get the same message.

I can contribute code if that helps, just wanted to get buy-in on the idea first.

Original issue: http://code.google.com/p/gogoprotobuf/issues/detail?id=18

From awalterschulze on July 25, 2014 14:59:38

Everything works, it just does not read very nicely

Original issue: http://code.google.com/p/gogoprotobuf/issues/detail?id=20

From [email protected] on July 24, 2014 18:31:14

What steps will reproduce the problem? 1. create embedded struct type like in the example
2. json.Marshal an instance of this type What is the expected output? What do you see instead? The expected output should be marshaled as if their inner exported fields were fields in the outer struct, as explained in the json documentation. Currently the tag json:"name" is generated, which produces a nested object in the json encoder. What version of the product are you using? On what operating system? current master on linux

Original issue: http://code.google.com/p/gogoprotobuf/issues/detail?id=19

From [email protected] on September 29, 2014 21:40:54

I have several times tried using custom types (mostly existing types from imported packages) with no luck. It always fails when I don't want to implement the proto.Marshaler interface, since I then have to wrap the type which bring (IMO) unnecessary complexity.

Since we can use a byte slice as a custom type, wouldn't it be perfectly logical to just use the existing go interface for marshalling to/from binary? ( http://golang.org/pkg/encoding/#BinaryMarshaler )

Also it should not be hard at all to implement, if you don't want this change maybe you could point me to the proper place for me to implement it locally?

Original issue: http://code.google.com/p/gogoprotobuf/issues/detail?id=28

From [email protected] on April 08, 2014 23:50:56

What steps will reproduce the problem? message A {
enum B {
C = 1;
}
required B b = 1 [(gogoproto.nullable) = false];
}

ERROR: field A.B cannot be nullable and be an enum type B which does not start with zero--gogo_out: protoc-gen-gogo: Plugin failed with status code 1.

But A.B is not nullable. defaultcheck.go has

        if gogoproto.IsNullable(field) {
            continue
        }
                    ...
        if len(enum.Value) == 0 || enum.Value[0].GetNumber() != 0 {
            fmt.Fprintf(os.Stderr, "ERROR: field %v.%v cannot be nullable and be an enum type %v which does not start with zero", generator.CamelCase(*msg.Name), generator.CamelCase(*field.Name), enum.GetName())

so it seems this case is never seen for nullable=true, and the error message is incorrect. How about this, adding "non-":

            fmt.Fprintf(os.Stderr, "ERROR: field %v.%v cannot be non-nullable and be an enum type %v which does not start with zero", generator.CamelCase(*msg.Name), generator.CamelCase(*field.Name), enum.GetName())

Original issue: http://code.google.com/p/gogoprotobuf/issues/detail?id=11

From jtolds on March 27, 2014 23:00:30

What steps will reproduce the problem? 1. build protocol buffers with gogoprotobuf defaults.
2. observe that the String() method works correctly on enums.
3. marshal the protocol buffer with an enum field What is the expected output? What do you see instead? you'd expect to see the name of the enum field (what goprotobuf does) as opposed to the integer value, but you see the integer value in the marshalled json instead. Please provide any additional information below. because of how the generator code is working, you get to pick. if you want the enum functionality to work correctly, you can disable the enum stringer

option (gogoproto.goproto_enum_stringer_all) = false;

but then you don't get the enum String() method.

generator.go has this code:

if gogoproto.IsGoEnumStringer(g.file.FileDescriptorProto, enum.EnumDescriptorProto) {
    g.P("func (x ", ccTypeName, ") String() string {")
    g.In()
    g.P("return ", g.Pkg["proto"], ".EnumName(", ccTypeName, "_name, int32(x))")
    g.Out()
    g.P("}")
}

if !gogoproto.IsGoEnumStringer(g.file.FileDescriptorProto, enum.EnumDescriptorProto) {
    g.P("func (x ", ccTypeName, ") MarshalJSON() ([]byte, error) {")
    g.In()
    g.P("return ", g.Pkg["proto"], ".MarshalJSONEnum(", ccTypeName, "_name, int32(x))")
    g.Out()
    g.P("}")
}

seems to me like you shouldn't have to choose, and you should get MarshalJSON anyway.

Original issue: http://code.google.com/p/gogoprotobuf/issues/detail?id=10

Is it reasonable to add a functionality to find out the offset of a field?

I want to write a protobuf {uint64, uint64, []byte} on to disk and mmap back a bytes field.

Thanks in advance!

From [email protected] on August 01, 2014 05:15:23

What steps will reproduce the problem? 1. Define protobuf message w/ a repeated field e.g. repeated int64 value_integer = 1;
2. Use generated code to instantiate a message w/ a value, serialize, save data to disk.
3. Specify field as packed repeated int64 value_integer = 1 [packed=true];
4. Regenerate protobuf code (no extensions), load saved data from disk, unmarshal, should work.
5. Activate sizer_all and unmarshaler_all extensions on message, regenerate protobuf code.
6. Load saved data from disk, unmarshal, should see missing data. What is the expected output? What do you see instead? Expect to see all original data, but unpacked data is missing. What version of the product are you using? On what operating system? Recent gogoprotobuf checkout, SHA d228c1a, on Linux. Please provide any additional information below. The generated Unmarshal code explicitly checks for wire type 2 w/ packed fields, and returns if wire type isn't 2. Ideally it would instead be structured like so:

if wireType == 2 {
.. do the regular packed field stuff ..
} else if wireType != {
return code_google_com_p_gogoprotobuf_proto.ErrWrongType
}
.. do the regular unpacked field stuff ..

This would allow packed repeated fields to fall back to default unmarshal code in cases where packing wasn't yet being used. The protobuf docs specify that this should be allowed, so as to support migration from unpacked to packed format for repeated fields w/o breaking wire compatibility: https://developers.google.com/protocol-buffers/docs/proto#options "In 2.3.0 and later, this change is safe, as parsers for packable fields will always accept both formats, ..."

Thanks!

Original issue: http://code.google.com/p/gogoprotobuf/issues/detail?id=21

From [email protected] on November 15, 2014 02:28:15

Protobufs with all optional fields end up becoming empty messages. http://godoc.org/code.google.com/p/gogoprotobuf/io is perfectly able to send these, as varint 0 and no further bytes. However, if the connection ends with such a message, NewDelimitedReader returns io.EOF, and the caller cannot know whether a message was unmarshaled or not.

Original issue: http://code.google.com/p/gogoprotobuf/issues/detail?id=31

From awalterschulze on October 08, 2013 17:33:59

Marshalling and unmarshalling requires unsafe when code is not generated. This removes the support goprotobuf had for appengine.
Use build constraints.

This might help https://github.com/davecheney/golang-crosscompile http://dave.cheney.net/2012/09/08/an-introduction-to-cross-compilation-with-go Also remember this:

SliceHeader is the runtime representation of a slice.
It cannot be used safely or portably.
It cannot be used safely or portably and its representation may
change in a later release.
Moreover, the Data field is not sufficient to guarantee the data
it references will not be garbage collected, so programs must keep
a separate, correctly typed pointer to the underlying data.
type SliceHeader struct {
Data uintptr
Len int

Original issue: http://code.google.com/p/gogoprotobuf/issues/detail?id=3

protobuf version 3 is out, being able to use gogoprotobuf with that would be nice.
https://github.com/google/protobuf/releases/tag/v3.0.0-alpha-2

From awalterschulze on October 24, 2013 14:14:01

What steps will reproduce the problem? 1. 2. 3. What is the expected output? What do you see instead? Please use labels and text to provide additional information.

Original issue: http://code.google.com/p/gogoprotobuf/issues/detail?id=5

hi~
how to use gogoprotobuf in golang standard pkg "net/rpc" and instead of its "gob".
i have tried to implement “gobServerCodec” and "ServerCodec interface", but seems no Decode() or Encode() method. so is there anyway to use "gogoprotobuf" instead of "gob".

From awalterschulze on September 07, 2014 16:50:28

Waiting for goprotobuf to do something

Original issue: http://code.google.com/p/gogoprotobuf/issues/detail?id=27

@awalterschulze I have diff against the pb.go file generated by the protoc compiler, with unsafe option turn on and off, but they look exactly the same. Can you highligh some pro and cons of using unsafe_marshaler.

thanks

From awalterschulze on October 14, 2013 16:53:35

Create another generation plugin for unmarshal merge.
Note: Do overwrite an already set pointer struct pointer, but overwrite its fields.

Original issue: http://code.google.com/p/gogoprotobuf/issues/detail?id=4

From awalterschulze on May 25, 2014 11:43:15

proto.Clone(pb Message) Message has not been tested with all gogoprotobuf scenarios.

If you are using this please comment so I can up the priority.

Original issue: http://code.google.com/p/gogoprotobuf/issues/detail?id=14

From [email protected] on November 10, 2014 20:28:08

Currently proto CONSTANTS_LIKE_THIS are being exposed as such within Go code instead of translating to camelCase like message names. This is causing golint to freak out.

An option (or change to the generated files) to generate golint-friendly code would be great.

Original issue: http://code.google.com/p/gogoprotobuf/issues/detail?id=30

//Unmarshals a buffer into a preallocated go friendly memory structure using the unsafe package.
func (this *m) UnmarshalInto(buf, mem []byte) error

This will be very unsafe and possibly go version dependant, but I think this will also be very fast.

This is some proof of concept work I did quite a while ago.
I just have not had the time to build this into a plugin.
I thought I'd better post it somewhere before I loose it.

package unmarshalinto

import (
    "bytes"
    "code.google.com/p/gogoprotobuf/proto"
    "code.google.com/p/gogoprotobuf/test"
    "math"
    "reflect"
    "testing"
    "unsafe"
)

/*
type MyExtendable struct {
    Field1           *int64                    `protobuf:"varint,1,opt" json:"Field1,omitempty"`
    XXX_extensions   map[int32]proto.Extension `json:"-"`
    XXX_unrecognized []byte                    `json:"-"`
}

extend MyExtendable {
    optional double FieldA = 100;
    optional NinOptNative FieldB = 101;
    optional NinEmbeddedStruct FieldC = 102;
}
*/

var (
    extMax = &test.MyExtendable{
        Field1: proto.Int64(math.MaxInt64),
    }
    extB = &test.NinOptNative{
        Field1: proto.Float64(math.MaxFloat64),
    }
)

func TestUnmarshalIntoMyExtendable(t *testing.T) {
    err := proto.SetExtension(extMax, test.E_FieldA, proto.Float64(math.MaxFloat64))
    if err != nil {
        panic(err)
    }
    err = proto.SetExtension(extMax, test.E_FieldB, extB)
    if err != nil {
        panic(err)
    }
    proto.GetRawExtension(extMax.XXX_extensions, 100)
    proto.GetRawExtension(extMax.XXX_extensions, 101)
    var msg *test.MyExtendable = nil
    initSize := int(unsafe.Offsetof(msg.XXX_unrecognized) + unsafe.Sizeof(msg.XXX_unrecognized))

    popSize := 1000

    buf := make([]byte, initSize+popSize)

    var offset int

    msg = (*test.MyExtendable)(unsafe.Pointer(&buf[0]))

    offset = initSize

    msg.Field1 = (*int64)(unsafe.Pointer(&buf[offset]))
    *msg.Field1 = math.MaxInt64
    offset += int(unsafe.Sizeof(*msg.Field1))

    //this is the key and wire type encoded as a varint
    buf[offset] = 161
    offset++
    buf[offset] = 6
    offset++

    a := (*float64)(unsafe.Pointer(&buf[offset]))
    offset += int(unsafe.Sizeof(*a))
    *a = math.MaxFloat64
    proto.SetRawExtension(msg, 100, buf[offset-10:offset])

    t.Logf("%v", buf)
    t.Logf("msg %#v", msg)
    printWords(t, buf)
    t.Logf("offset = %d offset/8 = %d", offset, offset/8)

    if err := msg.VerboseEqual(extMax); err != nil {
        panic(err)
    }
}

/*
message NinOptStruct {
    optional double Field1 = 1;
    optional float Field2 = 2;
    optional NidOptNative Field3 = 3;
    optional NinOptNative Field4 = 4;
    optional uint64 Field6 = 6;
    optional sint32 Field7 = 7;
    optional NidOptNative Field8 = 8;
    optional bool Field13 = 13;
    optional string Field14 = 14;
    optional bytes Field15 = 15;
}
*/

var (
    structMax2 = &test.NinOptStruct{
        Field1: proto.Float64(math.MaxFloat64),
        Field3: &test.NidOptNative{
            Field1: math.MaxFloat64 / 2,
        },
        Field4: &test.NinOptNative{
            Field2: proto.Float32(math.MaxFloat32 / 3),
        },
        Field6: proto.Uint64(math.MaxUint64 - 6),
    }
)

func TestUnmarshalIntoNinOptStruct2(t *testing.T) {
    var msg *test.NinOptStruct = nil
    initSize := int(unsafe.Offsetof(msg.XXX_unrecognized) + unsafe.Sizeof(msg.XXX_unrecognized))

    popSize := 1000

    buf := make([]byte, initSize+popSize)

    var offset int

    msg = (*test.NinOptStruct)(unsafe.Pointer(&buf[0]))

    offset = initSize

    msg.Field1 = (*float64)(unsafe.Pointer(&buf[offset]))
    *msg.Field1 = math.MaxFloat64
    offset += int(unsafe.Sizeof(*msg.Field1))

    msg.Field3 = (*test.NidOptNative)(unsafe.Pointer(&buf[offset]))
    msg.Field3.Field1 = math.MaxFloat64 / 2
    offset += int(unsafe.Sizeof(*msg.Field3))

    msg.Field4 = (*test.NinOptNative)(unsafe.Pointer(&buf[offset]))
    offset += int(unsafe.Sizeof(*msg.Field4))
    msg.Field4.Field2 = (*float32)(unsafe.Pointer(&buf[offset]))
    *msg.Field4.Field2 = math.MaxFloat32 / 3
    offset += int(unsafe.Sizeof(*msg.Field4.Field2))

    msg.Field6 = (*uint64)(unsafe.Pointer(&buf[offset]))
    *msg.Field6 = math.MaxUint64 - 6
    offset += int(unsafe.Sizeof(*msg.Field6))

    t.Logf("%v", buf)
    t.Logf("msg %#v", msg)
    printWords(t, buf)
    t.Logf("offset = %d offset/8 = %d", offset, offset/8)

    if err := msg.VerboseEqual(structMax2); err != nil {
        panic(err)
    }
}

var (
    structMax1 = &test.NinOptStruct{
        Field1: proto.Float64(math.MaxFloat64),
        Field3: &test.NidOptNative{},
    }
)

func TestUnmarshalIntoNinOptStruct1(t *testing.T) {
    var msg *test.NinOptStruct = nil
    initSize := int(unsafe.Offsetof(msg.XXX_unrecognized) + unsafe.Sizeof(msg.XXX_unrecognized))

    popSize := 1000

    buf := make([]byte, initSize+popSize)

    var offset int

    msg = (*test.NinOptStruct)(unsafe.Pointer(&buf[0]))

    offset = initSize

    msg.Field1 = (*float64)(unsafe.Pointer(&buf[offset]))
    *msg.Field1 = math.MaxFloat64
    offset += int(unsafe.Sizeof(*msg.Field1))

    msg.Field3 = (*test.NidOptNative)(unsafe.Pointer(&buf[offset]))
    offset += int(unsafe.Sizeof(*msg.Field3))

    t.Logf("%v", buf)
    t.Logf("msg %#v", msg)
    printWords(t, buf)
    t.Logf("offset = %d offset/8 = %d", offset, offset/8)

    if err := msg.VerboseEqual(structMax1); err != nil {
        panic(err)
    }
}

/*
message NinRepNative {
    repeated double Field1 = 1;
    repeated float Field2 = 2;
    repeated int32 Field3 = 3;
    repeated int64 Field4 = 4;
    repeated uint32 Field5 = 5;
    repeated uint64 Field6 = 6;
    repeated sint32 Field7 = 7;
    repeated sint64 Field8 = 8;
    repeated fixed32 Field9 = 9;
    repeated sfixed32 Field10 = 10;
    repeated fixed64 Field11 = 11;
    repeated sfixed64 Field12 = 12;
    repeated bool Field13 = 13;
    repeated string Field14 = 14;
    repeated bytes Field15 = 15;
}
*/

var (
    repMax = &test.NinRepNative{
        Field1:  []float64{math.MaxFloat64},
        Field2:  []float32{math.MaxFloat32, math.MaxFloat32 / 2},
        Field3:  []int32{math.MaxInt32, math.MaxInt32 / 2, math.MaxInt32 / 3},
        Field14: []string{"ABCD", "CDEF"},
        Field15: [][]byte{{1, 2, 3, 4}, {5, 6, 7}},
    }
)

func TestUnmarshalIntoNinRepNative(t *testing.T) {
    var msg *test.NinRepNative = nil
    initSize := int(unsafe.Offsetof(msg.XXX_unrecognized) + unsafe.Sizeof(msg.XXX_unrecognized))

    popSize := 1000

    buf := make([]byte, initSize+popSize)

    var offset int

    msg = (*test.NinRepNative)(unsafe.Pointer(&buf[0]))

    offset = initSize

    v1 := []float64{math.MaxFloat64}
    f1 := (*reflect.SliceHeader)(unsafe.Pointer(&buf[int(unsafe.Offsetof(msg.Field1))]))
    f1.Data = uintptr(unsafe.Pointer(&buf[offset]))
    f1.Len = 0
    f1.Cap = len(v1)
    for _, v := range v1 {
        msg.Field1 = append(msg.Field1, v)
        offset += int(unsafe.Sizeof(v))
    }

    v2 := []float32{math.MaxFloat32, math.MaxFloat32 / 2}
    f2 := (*reflect.SliceHeader)(unsafe.Pointer(&buf[int(unsafe.Offsetof(msg.Field2))]))
    f2.Data = uintptr(unsafe.Pointer(&buf[offset]))
    f2.Len = 0
    f2.Cap = len(v2)
    for _, v := range v2 {
        msg.Field2 = append(msg.Field2, v)
        offset += int(unsafe.Sizeof(v))
    }

    v3 := []int32{math.MaxInt32, math.MaxInt32 / 2, math.MaxInt32 / 3}
    f3 := (*reflect.SliceHeader)(unsafe.Pointer(&buf[int(unsafe.Offsetof(msg.Field3))]))
    f3.Data = uintptr(unsafe.Pointer(&buf[offset]))
    f3.Len = 0
    f3.Cap = len(v3)
    for _, v := range v3 {
        msg.Field3 = append(msg.Field3, v)
        offset += int(unsafe.Sizeof(v))
    }

    v14 := []string{"ABCD", "CDEF"}
    f14 := (*reflect.SliceHeader)(unsafe.Pointer(&buf[int(unsafe.Offsetof(msg.Field14))]))
    f14.Data = uintptr(unsafe.Pointer(&buf[offset]))
    f14.Len = len(v14)
    f14.Cap = len(v14)
    n := len(v14) * 16
    elem_offset := offset + n
    for _, v := range v14 {
        r := (*reflect.StringHeader)(unsafe.Pointer(&buf[offset]))
        r.Data = uintptr(unsafe.Pointer(&buf[elem_offset]))
        r.Len = len(v)
        copy(buf[elem_offset:], v)
        elem_offset += len(v)
        offset += 16
    }
    offset = elem_offset

    v15 := [][]byte{{1, 2, 3, 4}, {5, 6, 7}}
    f15 := (*reflect.SliceHeader)(unsafe.Pointer(&buf[int(unsafe.Offsetof(msg.Field15))]))
    f15.Data = uintptr(unsafe.Pointer(&buf[offset]))
    f15.Len = len(v15)
    f15.Cap = len(v15)
    n15 := len(v15) * 24
    elem_offset15 := offset + n15
    for _, v := range v15 {
        r := (*reflect.SliceHeader)(unsafe.Pointer(&buf[offset]))
        r.Data = uintptr(unsafe.Pointer(&buf[elem_offset15]))
        r.Len = len(v)
        r.Cap = len(v)
        copy(buf[elem_offset15:], v)
        elem_offset15 += len(v)
        offset += 24
    }

    t.Logf("%v", buf)
    t.Logf("msg %#v", msg)
    printWords(t, buf)
    t.Logf("offset = %d offset/8 = %d", offset, offset/8)

    if err := msg.VerboseEqual(repMax); err != nil {
        panic(err)
    }
}

/*
type NidOptNative struct {
    Field1           float64 `protobuf:"fixed64,1,opt" json:"Field1"`
    Field2           float32 `protobuf:"fixed32,2,opt" json:"Field2"`
    Field3           int32   `protobuf:"varint,3,opt" json:"Field3"`
    Field4           int64   `protobuf:"varint,4,opt" json:"Field4"`
    Field5           uint32  `protobuf:"varint,5,opt" json:"Field5"`
    Field6           uint64  `protobuf:"varint,6,opt" json:"Field6"`
    Field7           int32   `protobuf:"zigzag32,7,opt" json:"Field7"`
    Field8           int64   `protobuf:"zigzag64,8,opt" json:"Field8"`
    Field9           uint32  `protobuf:"fixed32,9,opt" json:"Field9"`
    Field10          int32   `protobuf:"fixed32,10,opt" json:"Field10"`
    Field11          uint64  `protobuf:"fixed64,11,opt" json:"Field11"`
    Field12          int64   `protobuf:"fixed64,12,opt" json:"Field12"`
    Field13          bool    `protobuf:"varint,13,opt" json:"Field13"`
    Field14          string  `protobuf:"bytes,14,opt" json:"Field14"`
    Field15          []byte  `protobuf:"bytes,15,opt" json:"Field15"`
    XXX_unrecognized []byte  `json:"-"`
}
*/

var (
    nidMax = &test.NidOptNative{
        Field1:           math.MaxFloat64 - 1,
        Field2:           math.MaxFloat32 - 2,
        Field3:           math.MaxInt32 - 3,
        Field4:           math.MaxInt64 - 4,
        Field5:           math.MaxUint32 - 5,
        Field6:           math.MaxUint64 - 6,
        Field7:           math.MaxInt32 - 7,
        Field8:           math.MaxInt64 - 8,
        Field9:           math.MaxUint32 - 9,
        Field10:          math.MaxInt32 - 10,
        Field11:          math.MaxUint64 - 11,
        Field12:          math.MaxInt64 - 12,
        Field13:          true,
        Field14:          "ABCDEFGHI",
        Field15:          []byte{1, 2, 3, 5, 4, 5, 6, 7, 8},
        XXX_unrecognized: []byte{255, 254, 253, 1, 252, 251, 250, 249, 248},
    }
)

func TestUnmarshalIntoNidOptNative(t *testing.T) {
    s := []byte("ABCDEFGHI")
    data := []byte{1, 2, 3, 5, 4, 5, 6, 7, 8}
    xxx := []byte{255, 254, 253, 1, 252, 251, 250, 249, 248}

    popSize := len(s)
    popSize += len(data)
    popSize += len(xxx)

    var msg *test.NidOptNative = nil

    fieldPointerSize := int(unsafe.Offsetof(msg.XXX_unrecognized) + unsafe.Sizeof(msg.XXX_unrecognized))
    //fieldPointerSize := (11 + 2 + 3 + 3) * 8

    buf := make([]byte, fieldPointerSize+popSize)

    var offset int

    msg = (*test.NidOptNative)(unsafe.Pointer(&buf[0]))

    msg.Field1 = math.MaxFloat64 - 1
    msg.Field2 = math.MaxFloat32 - 2
    msg.Field3 = math.MaxInt32 - 3
    msg.Field4 = math.MaxInt64 - 4
    msg.Field5 = math.MaxUint32 - 5
    //see how 4 bytes are left empty to better align the bytes
    msg.Field6 = math.MaxUint64 - 6
    msg.Field7 = math.MaxInt32 - 7
    //and again 4 bytes are left empty for alignment
    msg.Field8 = math.MaxInt64 - 8
    msg.Field9 = math.MaxUint32 - 9
    msg.Field10 = math.MaxInt32 - 10
    msg.Field11 = math.MaxUint64 - 11
    msg.Field12 = math.MaxInt64 - 12
    msg.Field13 = true
    //the StringHeader is stored here

    offset = fieldPointerSize

    r := (*reflect.StringHeader)(unsafe.Pointer(&buf[int(unsafe.Offsetof(msg.Field14))]))
    r.Data = uintptr(unsafe.Pointer(&buf[offset]))
    r.Len = len(s)

    copy(buf[offset:], s)
    offset += len(s)

    b := (*reflect.SliceHeader)(unsafe.Pointer(&buf[int(unsafe.Offsetof(msg.Field15))]))
    b.Data = uintptr(unsafe.Pointer(&buf[offset]))
    b.Len = len(data)
    b.Cap = len(data)

    copy(buf[offset:], data)
    offset += len(data)

    x := (*reflect.SliceHeader)(unsafe.Pointer(&buf[int(unsafe.Offsetof(msg.XXX_unrecognized))]))
    x.Data = uintptr(unsafe.Pointer(&buf[offset]))
    x.Len = len(xxx)
    x.Cap = len(xxx)

    copy(buf[offset:], xxx)
    offset += len(xxx)

    t.Logf("%v", buf)
    t.Logf("msg %#v", msg)
    printWords(t, buf)
    t.Logf("offset = %d offset/8 = %d", offset, offset/8)

    if err := msg.VerboseEqual(nidMax); err != nil {
        panic(err)
    }
}

/*
type NinOptNative struct {
    Field1           *float64 `protobuf:"fixed64,1,opt" json:"Field1,omitempty"`
    Field2           *float32 `protobuf:"fixed32,2,opt" json:"Field2,omitempty"`
    Field3           *int32   `protobuf:"varint,3,opt" json:"Field3,omitempty"`
    Field4           *int64   `protobuf:"varint,4,opt" json:"Field4,omitempty"`
    Field5           *uint32  `protobuf:"varint,5,opt" json:"Field5,omitempty"`
    Field6           *uint64  `protobuf:"varint,6,opt" json:"Field6,omitempty"`
    Field7           *int32   `protobuf:"zigzag32,7,opt" json:"Field7,omitempty"`
    Field8           *int64   `protobuf:"zigzag64,8,opt" json:"Field8,omitempty"`
    Field9           *uint32  `protobuf:"fixed32,9,opt" json:"Field9,omitempty"`
    Field10          *int32   `protobuf:"fixed32,10,opt" json:"Field10,omitempty"`
    Field11          *uint64  `protobuf:"fixed64,11,opt" json:"Field11,omitempty"`
    Field12          *int64   `protobuf:"fixed64,12,opt" json:"Field12,omitempty"`
    Field13          *bool    `protobuf:"varint,13,opt" json:"Field13,omitempty"`
    Field14          *string  `protobuf:"bytes,14,opt" json:"Field14,omitempty"`
    Field15          []byte   `protobuf:"bytes,15,opt" json:"Field15,omitempty"`
    XXX_unrecognized []byte   `json:"-"`
}
*/

var (
    ninMax = &test.NinOptNative{
        Field1:           proto.Float64(math.MaxFloat64 - 1),
        Field2:           proto.Float32(math.MaxFloat32 - 2),
        Field3:           proto.Int32(math.MaxInt32 - 3),
        Field4:           proto.Int64(math.MaxInt64 - 4),
        Field5:           proto.Uint32(math.MaxUint32 - 5),
        Field6:           proto.Uint64(math.MaxUint64 - 6),
        Field7:           proto.Int32(math.MaxInt32 - 7),
        Field8:           proto.Int64(math.MaxInt64 - 8),
        Field9:           proto.Uint32(math.MaxUint32 - 9),
        Field10:          proto.Int32(math.MaxInt32 - 10),
        Field11:          proto.Uint64(math.MaxUint64 - 11),
        Field12:          proto.Int64(math.MaxInt64 - 12),
        Field13:          proto.Bool(true),
        Field14:          proto.String("ABCDEFGHI"),
        Field15:          []byte{1, 2, 3, 5, 4, 5, 6, 7, 8},
        XXX_unrecognized: []byte{255, 254, 253, 1, 252, 251, 250, 249, 248},
    }
)

//slice of bytes takes 3*8 bytes, one for pointer, one for len and one for cap, the sliceheader is stored inline.
//string takes 8 bytes, since this is only a pointer to the string header, unlike the slice header
//seems all other native pointers also take 8 bytes

func TestUnmarshalIntoNinOptNative(t *testing.T) {
    s := []byte("ABCDEFGHI")
    data := []byte{1, 2, 3, 5, 4, 5, 6, 7, 8}
    xxx := []byte{255, 254, 253, 1, 252, 251, 250, 249, 248}

    var msg *test.NinOptNative = nil

    popSize := int(unsafe.Sizeof(*msg.Field1)) + int(unsafe.Sizeof(*msg.Field2)) + int(unsafe.Sizeof(*msg.Field3)) +
        int(unsafe.Sizeof(*msg.Field4)) + int(unsafe.Sizeof(*msg.Field5)) + int(unsafe.Sizeof(*msg.Field6)) +
        int(unsafe.Sizeof(*msg.Field7)) + int(unsafe.Sizeof(*msg.Field8)) + int(unsafe.Sizeof(*msg.Field9)) +
        int(unsafe.Sizeof(*msg.Field10)) + int(unsafe.Sizeof(*msg.Field11)) + int(unsafe.Sizeof(*msg.Field12)) + int(unsafe.Sizeof(*msg.Field13))
    popSize += 16 + len(s)
    popSize += len(data)
    popSize += len(xxx)

    fieldPointerSize := int(unsafe.Offsetof(msg.XXX_unrecognized) + unsafe.Sizeof(msg.XXX_unrecognized))
    //fieldPointerSize := (14 + 3 + 3) * 8

    buf := make([]byte, fieldPointerSize+popSize)

    var offset int

    msg = (*test.NinOptNative)(unsafe.Pointer(&buf[0]))

    offset = fieldPointerSize

    msg.Field1 = (*float64)(unsafe.Pointer(&buf[offset]))
    *msg.Field1 = math.MaxFloat64 - 1
    offset += int(unsafe.Sizeof(*msg.Field1))

    msg.Field2 = (*float32)(unsafe.Pointer(&buf[offset]))
    *msg.Field2 = math.MaxFloat32 - 2
    offset += int(unsafe.Sizeof(*msg.Field2))

    msg.Field3 = (*int32)(unsafe.Pointer(&buf[offset]))
    *msg.Field3 = math.MaxInt32 - 3
    offset += int(unsafe.Sizeof(*msg.Field3))

    msg.Field4 = (*int64)(unsafe.Pointer(&buf[offset]))
    *msg.Field4 = math.MaxInt64 - 4
    offset += int(unsafe.Sizeof(*msg.Field4))

    msg.Field5 = (*uint32)(unsafe.Pointer(&buf[offset]))
    *msg.Field5 = math.MaxUint32 - 5
    offset += int(unsafe.Sizeof(*msg.Field5))

    msg.Field6 = (*uint64)(unsafe.Pointer(&buf[offset]))
    *msg.Field6 = math.MaxUint64 - 6
    offset += int(unsafe.Sizeof(*msg.Field6))

    msg.Field7 = (*int32)(unsafe.Pointer(&buf[offset]))
    *msg.Field7 = math.MaxInt32 - 7
    offset += int(unsafe.Sizeof(*msg.Field7))

    msg.Field8 = (*int64)(unsafe.Pointer(&buf[offset]))
    *msg.Field8 = math.MaxInt64 - 8
    offset += int(unsafe.Sizeof(*msg.Field8))

    msg.Field9 = (*uint32)(unsafe.Pointer(&buf[offset]))
    *msg.Field9 = math.MaxUint32 - 9
    offset += int(unsafe.Sizeof(*msg.Field9))

    msg.Field10 = (*int32)(unsafe.Pointer(&buf[offset]))
    *msg.Field10 = math.MaxInt32 - 10
    offset += int(unsafe.Sizeof(*msg.Field10))

    msg.Field11 = (*uint64)(unsafe.Pointer(&buf[offset]))
    *msg.Field11 = math.MaxUint64 - 11
    offset += int(unsafe.Sizeof(*msg.Field11))

    msg.Field12 = (*int64)(unsafe.Pointer(&buf[offset]))
    *msg.Field12 = math.MaxInt64 - 12
    offset += int(unsafe.Sizeof(*msg.Field12))

    msg.Field13 = (*bool)(unsafe.Pointer(&buf[offset]))
    *msg.Field13 = true
    offset += int(unsafe.Sizeof(*msg.Field13))
    //offset += 8 //this might smaller than eight, for example one byte seems enough

    //stores a string header here, which contains a pointer and a length = 16 bytes
    msg.Field14 = (*string)(unsafe.Pointer(&buf[offset]))
    //access the string header
    r := (*reflect.StringHeader)(unsafe.Pointer(&buf[offset]))
    offset += int(unsafe.Sizeof(*msg.Field14))
    //set the pointer to the string
    r.Data = uintptr(unsafe.Pointer(&buf[offset]))
    //set the string length
    r.Len = len(s)
    //copy the string into the buffer
    copy(buf[offset:], s)
    offset += len(s)

    msg.Field15 = *(*[]byte)(unsafe.Pointer(&buf[int(unsafe.Offsetof(msg.Field15))]))
    b := (*reflect.SliceHeader)(unsafe.Pointer(&buf[int(unsafe.Offsetof(msg.Field15))]))
    b.Data = uintptr(unsafe.Pointer(&buf[offset]))
    b.Len = len(data)
    b.Cap = b.Len
    copy(buf[offset:], data)
    offset += len(data)

    msg.XXX_unrecognized = *(*[]byte)(unsafe.Pointer(&buf[int(unsafe.Offsetof(msg.XXX_unrecognized))]))
    x := (*reflect.SliceHeader)(unsafe.Pointer(&buf[int(unsafe.Offsetof(msg.XXX_unrecognized))]))
    x.Data = uintptr(unsafe.Pointer(&buf[offset]))
    x.Len = len(xxx)
    x.Cap = b.Len
    copy(buf[offset:], xxx)
    offset += len(xxx)

    t.Logf("%v", buf)
    t.Logf("msg %#v", msg)
    printWords(t, buf)
    t.Logf("offset = %d offset/8 = %d", offset, offset/8)

    if err := msg.VerboseEqual(ninMax); err != nil {
        panic(err)
    }
}

func printWords(t *testing.T, buf []byte) {
    zeros := []byte{0, 0, 0, 0, 0, 0, 0, 0}
    size := len(buf) / 8
    for i := 0; i < size; i++ {
        if !bytes.Equal(buf[i*8:(i+1)*8], zeros) {
            t.Logf("%d:%v", i+1, buf[i*8:(i+1)*8])
        }
    }
    t.Logf("last:%v", buf[size*8:])
}

This is currently done for JSON, but not YAML.

https://github.com/gogo/protobuf/blob/master/plugin/defaultcheck/defaultcheck.go#L100 warns:

WARNING: field NodeStatus.StoreIds is a repeated non-nullable native type, nullable=false has no effect

however, this is not true, because nullable=false does have an effect. Removing it:

diff --git a/proto/status.proto b/proto/status.proto
index d735e64..1465466 100644
--- a/proto/status.proto
+++ b/proto/status.proto
@@ -45,7 +45,7 @@ message StoreStatus {
 // node.
 message NodeStatus {
   optional NodeDescriptor desc = 1 [(gogoproto.nullable) = false];
-  repeated int32 store_ids = 2 [(gogoproto.nullable) = false, (gogoproto.customname) = "StoreIDs"];
+  repeated int32 store_ids = 2 [(gogoproto.customname) = "StoreIDs"];
   optional int32 range_count = 3 [(gogoproto.nullable) = false];
   optional int64 started_at = 4 [(gogoproto.nullable) = false];
   optional int64 updated_at = 5 [(gogoproto.nullable) = false];

causes:

diff --git a/proto/status.pb.go b/proto/status.pb.go
index 0947198..0468495 100644
--- a/proto/status.pb.go
+++ b/proto/status.pb.go
@@ -96,7 +96,7 @@ func (m *StoreStatus) GetAvailableRangeCount() int32 {
 // node.
 type NodeStatus struct {
    Desc                 NodeDescriptor `protobuf:"bytes,1,opt,name=desc" json:"desc"`
-   StoreIDs             []int32        `protobuf:"varint,2,rep,name=store_ids" json:"store_ids"`
+   StoreIDs             []int32        `protobuf:"varint,2,rep,name=store_ids" json:"store_ids,omitempty"`
    RangeCount           int32          `protobuf:"varint,3,opt,name=range_count" json:"range_count"`
    StartedAt            int64          `protobuf:"varint,4,opt,name=started_at" json:"started_at"`
    UpdatedAt            int64          `protobuf:"varint,5,opt,name=updated_at" json:"updated_at"`

Notice that JSON handling has changed - StoreIDs has become nullable in JSON.

When setting nullable = false for a field, we can't say whether optional field is set or not. But we could generate additional field has_<field_name> bool that says whether field is set or not.

From awalterschulze on May 07, 2014 21:17:58

proto.Equal(a, b Message) bool has not been tested with all gogoprotobuf scenarios.

If you are using this please comment so I can up the priority.

Original issue: http://code.google.com/p/gogoprotobuf/issues/detail?id=13

From [email protected] on January 10, 2014 02:07:11

Using this input file as one.proto, with the commands listed.

Not sure what the best possible behavior would be, but the current behavior leaves the user to debugging things on their own. Would anyone ever want to use marshaler but provide their own custom Size method?

/*

protoc -I$GOPATH/src -I. --gogo_out=. one.proto

$ go build one.pb.go

command-line-arguments

./one.pb.go:65: m.Size undefined (type *Dirent has no field or method Size)

*/
package one;

import "code.google.com/p/gogoprotobuf/gogoproto/gogo.proto";

// uncommenting the below fixes the problem
//option (gogoproto.sizer_all) = true;
option (gogoproto.marshaler_all) = true;

message Dirent {
required uint64 inode = 1 [(gogoproto.nullable) = false];
enum Type {
FILE = 1;
DIR = 2;
};
required Type type = 2 [(gogoproto.nullable) = false];
}

Original issue: http://code.google.com/p/gogoprotobuf/issues/detail?id=7

From awalterschulze on May 06, 2014 21:17:54

I suspect this feature is very rarely used.

message_set_wire_format option will have some undefined conflicts with certain extensions.

If you would like this to be fixed please comment.

Thank you.

Original issue: http://code.google.com/p/gogoprotobuf/issues/detail?id=12

From awalterschulze on May 28, 2014 14:35:56

What steps will reproduce the problem? 1. go tool vet . 2. 3. What is the expected output? What do you see instead? Expect no output, but I get
proto/pointer_unsafe_gogo.go:55: possible misuse of unsafe.Pointer
proto/pointer_unsafe_gogo.go:56: possible misuse of unsafe.Pointer Please use labels and text to provide additional information.

Original issue: http://code.google.com/p/gogoprotobuf/issues/detail?id=15

The generated marshalling code is building messages out of order. While still usable, this doesn't conform to spec.

Issue is most likely at https://github.com/gogo/protobuf/blob/master/plugin/marshalto/marshalto.go#L319, and message.Field just needs to be sorted by tag number

From [email protected] on August 15, 2014 21:06:25

What steps will reproduce the problem? 1. Test using github.com/satori/go.uuid.UUID as a custom type (works)
2. Instead use github.com/dynamic-design/go.uuid.UUID which is a fork without much changes
3. Will get "protoc-gen-gogo: error:bad Go source code was generated: 79:25: expected ';', found '-' (and 2 more errors)"
4. Simply removing the dash making it github.com/dynamicdesign/go.uuid makes it work (but wrong import path ofc)

Original issue: http://code.google.com/p/gogoprotobuf/issues/detail?id=25

Using a custom type with the unmarshaler plugin results in code that does not compile. I've created a tiny repro here: https://github.com/tamird/gogoproto_customtype

From awalterschulze on October 31, 2013 09:48:47

Currently it is zero

Original issue: http://code.google.com/p/gogoprotobuf/issues/detail?id=6