Work In Progress: proto3 and some other issues. about protobuf HOT 21 CLOSED

Having played with proto3 for a few hours, it seems that most of the extensions in gogoprotobuf are not really needed anymore. Would you entertain the idea of splitting out a project that does nothing but codegens proto.Marshalers to accelerate goprotobuf, without trying to replace any of the code there? That approach wouldn't need the "proto" library duplication etc, and proto2 compat wouldn't be risked because the new stuff would be in a separate tree.

from protobuf.

I will definitely be adding proto3 support.
I am just waiting for the documentation to come out, so that I don't make any premature mistakes.

from protobuf.

My perspective, of proto3 making gogoproto extensions mostly unnecessarily, is probably heavily biased by the set of extensions I use. I just catalogued my fairly-big project, here's what I use:

Global:
option (gogoproto.marshaler_all) = true;
option (gogoproto.onlyone) = true;
option (gogoproto.sizer_all) = true;
option (gogoproto.unmarshaler_all) = true;

Per-field:
(gogoproto.customname) = "Size_"
(gogoproto.customtype) = "foo"
(gogoproto.embed) = true
(gogoproto.nullable) = false

So let's first cover these..

1. enable code gen; kinda implicit, that's what i wanted in the first place
   option (gogoproto.marshaler_all) = true;
   option (gogoproto.sizer_all) = true;
   option (gogoproto.unmarshaler_all) = true;
2. onlyone:
   option (gogoproto.onlyone) = true;
   oneof is upstream now: https://developers.google.com/protocol-buffers/docs/proto#oneof
3. Size method collision workaround; this really should never be needed in the first place
   (gogoproto.customname) = "Size_"
4. custom type: this never really worked as well as I wanted, it's just sugar over a []byte that gets length-prefixed
   (gogoproto.customtype) = "foo"
5. embed: i use this only to avoid a pointer indirection for unions, and those i'll redo with oneof
   (gogoproto.embed) = true
6. nullable: used to avoid the *string plague, but proto3 removes that; only meaningful for sub-messages now
   (gogoproto.nullable) = false

For the extensions that I don't personally use, at least some of them seem less useful now:

1. goproto_getters: proto3 doesn't generate getters
2. goproto_unrecognized: proto3 no longer has XXX_unrecognized
3. stringer: generated by proto3, though not code gen
4. jsontag: proto3 has a canonical json representation

By two proto libraries, I do mean goprotobuf/proto and gogoprotobuf/proto. To use gogoproto and random projects regularly, I need both in my GOPATH. Having both of them means that things like goimports become timebombs; I can't just type proto.Marshal in my code and let goimports figure it out, because if it picks the wrong one, the errors that result are really not obvious; things often compile but misbehave. This has already bit me several times.

Plugins as in golang/protobuf#7 looks really promising.

I would really like to see a future where I can get code-gen speedup, but still use goprotobuf/proto and be compatible with the mainstream protobuf-using Go programmers.

Maybe many of the gogoprotobuf plugins can be implemented with the --g_out=plugins way, with the end result being more modular.

from protobuf.

I am looking at the diff between the two proto folders.

• extensions_bytes
• some enum stuff
• custom type
• encoding and decoding stuff

Most of these are not necessary if you generate code.
So there could be a world where I say I give a warning when you try to use these without generating code.

Maybe I should also start collecting some issues together and start planning a branch with some breaking changes that make things simpler.

But I am going to be busy this week.
So I'll keep thinking about this and get back to you asap.

from protobuf.

I am going to start adding proto3 support on a branch called proto3.
I don't want to support it officially until the documentation comes out.
After this is working we can talk about further possibly breaking changes.
How does that sound?

from protobuf.

proto3 support is coming along quite nicely on the proto3 branch

from protobuf.

Currently there are some tests failing because of this issue
golang/protobuf#18

from protobuf.

I have caught up to golang/protobuf and merged grpc.
golang/protobuf@c22ae3c
This bug occurs when empty byte slices are used as map values.
golang/protobuf#20

from protobuf.

golang/protobuf#21

from protobuf.

issue 20 and 21 have been closed and merged into the proto3 branch.

from protobuf.

Ok I think it should be easy to add another binary that simply takes the parsed fileDescriptorSet and edits it the way you want the defaults and then simply does all the same code gen.
What do you think?

from protobuf.

So it basically has a lot of implicit behaviour.

from protobuf.

We could then make another option/binary that does the same but simply imports goprotobuf instead of gogoprotobuf.
We could then make sure that for these implicit behaviours (options) has no need of the edits made in gogoprotobuf/proto.

from protobuf.

Sorry for the super long delay.

from protobuf.

Hello?

from protobuf.

I contain this issue to proto3 support.
The other issues I think are similar to the solution proposed in:
#39

from protobuf.

@tv42 would proto3 support +
#55 : An extension to allow you to not import gogoprotobuf +
#39 : protoc-gen-gogovanity which adds extensions for you so you don't have to +
#56 : Renaming the field Size to Size_ for you just like golang/protobuf renames the String field to String_
Resolve your issue?

from protobuf.

Please check out the size fieldname fix on the sizeunderscore branch.

from protobuf.

#55 is also in a reviewable state

from protobuf.

as is issue 56

from protobuf.

I am going to open a new issue for proto3 support.
I have merged all these issues into the proto3 branch.
If you have more please open another issue.

protoc -go_fast=. myproto.proto will probably be what you want to use know.
See the readme in the proto3 branch.

from protobuf.