elixir-protobuf / protobuf Goto Github PK

View Code? Open in Web Editor NEW

790.0 790.0 140.0 1.11 MB

A pure Elixir implementation of Google Protobuf.

Home Page: https://hexdocs.pm/protobuf/readme.html

License: MIT License

Elixir 99.72% HTML 0.25% Shell 0.03%

elixir google-protocol-buffers protobuf protobuf-elixir protoc protocol-buffers serialization

protobuf's People

Contributors

Stargazers

Watchers

Forkers

arnaud-lb princemaple hayesgm vonmixer cjab amatalai chansuke davidmikesimon peddle ch4s3 hotogwc zentetsukenz falood qinix llxff quantumghost 2nd gialib renesugar renderedtext xiaobin0860 maartenvanvliet liuzf1986 herlitzd jayh5 nktks bmarkons edwardzhou lmarlow tyrchen hassox ccpgames jackmarchant matehat bblaszkow06 marltake stjordanis anoskov connorjacobsen demilletech fbergero rolandtritsch kr00lix jeanparpaillon jared-mackey mikalv mergefailure haljin pharosproduction artemeff sizigi brexhq coingaming cswen xenomorphtech britto actionsprout bamorim perezd zetaron shengwu xinz maennchen andrelip sheriefalaa kbaird fukuru wingyplus flatiron-labs kianmeng offgridelectric ewestern strogo shoz-f jechol kfabryczny kwanta-cante redink ericmj ktec abc3 mrjoelkemp sunilkelkar tubitv wojtekmach mikebveil zolakeith nasrulgunawan arctic-grpc preveil supersimple nallwhy cyberflamego maysgar bceskavich danmarcab cytro54 millionintegrals davelooi royal-markets

protobuf's Issues

Feature request: option for different package name (module name) in elixir

Like this:

package google.bigtable.v1;

option go_package = "google.golang.org/genproto/googleapis/bigtable/v1;bigtable";
option java_package = "com.google.bigtable.v1";

What's providing protoc-gen-elixir ?

From the readme:

Install protoc plugin protoc-gen-elixir for Elixir . NOTE: You have to make sure protoc-gen-elixir(this name is important) is in your PATH.

What project is providing protoc-gen-elixir ? Is it gRPC, this project, or an other independent project ?

Cannot encode enum fields with atoms as values

Given such proto file:

enum Ble {
  a = 0;
  test = 1;
}

message WithEnum {
  Ble stuff = 1;
}

then I cannot do:

iex(11)> Api.WithEnum.new(stuff: :test) |> Api.WithEnum.encode
** (Protobuf.InvalidError) Ngcp.MobilityService.Api.WithEnum#stuff is invalid!
    lib/protobuf/validator.ex:5: Protobuf.Validator.validate!/1
    lib/protobuf/encoder.ex:9: Protobuf.Encoder.encode/2

but have to instead:

iex(10)> Api.WithEnum.new(stuff: Api.Ble.value(:test)) |> Api.WithEnum.encode
<<8, 1>>

I think it would be much more convenient to just use atom values directly as field values for messages containing enums. I'd be more than willing to make a PR that does that if you'd just point me in the right direction. :)

Proto file has this line: import "google/api/annotations.proto", how to import?

This is my error when I run protoc --elixir_out=./lib */*.proto
lib/rpc.proto:3:1: Import "google/api/annotations.proto" was not found or had errors.

Error trying to find proto file

I have my proto file in on the top level of my directory. When I run this: protoc --elixir_out=./lib rpc.proto, I get the following error:

Please specify a program using absolute path or make sure the program is available in your PATH system variable
--elixir_out: protoc-gen-elixir: Plugin failed with status code 1.

I've tried many variations of appending to my path without luck. The latest one is: export PATH=/Users/emmyalmaraz/.mix/escripts/bin:$PATH

Many versions are not published to hex.

protoc --elixir_out=./lib helloworld.proto

Decoder fails when some fields are empty

Great library! Thanks. I have a question though.

We are experiencing errors like the following when trying to decode messages that do not include all of the fields in a given protobuf (proto3). It is my understanding that proto3 should default to various "nully" values depending on the type of the attribute.

%UndefinedFunctionError{arity: 1, function: :key, message: nil, module: ErrorCode, reason: nil}

For example the following protobuf fails to decode incoming messages that do not include an a value or a b value, etc.

  defmodule PayInvoiceResponse do
    @moduledoc false
    use Protobuf, syntax: :proto3

    @type t :: %__MODULE__{
            a: Error.t() | nil,
            b: String.t(),
            c: integer,
            d: binary
          }
    defstruct [:a, :b, :c, :d]

    field :a, 2, type: Error
    field :b, 3, type: :string
    field :c, 4, type: :int64
    field :d, 5, type: :bytes
  end

Are we supposed to be using the :use_default option? Thanks!

Error when parsing services without an explicit package name.

Similar to #27 - I'm getting a crash when I try to compile the following proto file:

syntax = "proto3";

message RenderRequest {
}

message RenderResult {
    string html = 1;
}

service ElmRender {
    rpc Render(RenderRequest) returns (RenderResult);
}

If I add package elm_render; it works fine.

Compiling failures with enum

== Compilation error in file lib/protos/content/service.pb.ex ==
** (UndefinedFunctionError) function Content.Foo.key/1 is undefined (module Content.Foo is not available)
    Content.ThumbnailSpritesType.key(0)
    lib/protobuf/dsl.ex:32: anonymous fn/2 in Protobuf.DSL."MACRO-__before_compile__"/2
    (elixir) lib/enum.ex:1948: Enum."-reduce/3-lists^foldl/2-0-"/3
    expanding macro: Protobuf.DSL.__before_compile__/1
    lib/protos/content/service.pb.ex:254: Content.Bar (module)`

It seems this is caused by this PR. I guess the reason is Content.Foo is defined at another file, and with this PR's change, Content.Foo is not compiled when compiling Content.Bar.

Nested messages typespecs should include nil as possible value

Sometimes when you do b.a || true in your code (where b and a are nested messages) dialyzer returns following warning:
Guard test _@3::#{'__struct__':='Elixir.Example.A', 'text':=binary()} =:= 'false' can never succeed

Repo to reproduce issue https://github.com/amatalai/protobuf_nil_example

Should a message of default values be removed after encode?

For example, the proto is:

message User { uint32 id = 1; }

A User of id 0 is as if a non-specified one after encode:

iex> u = User.new id: 0
%User{id: 0}
iex> User.encode u
""
iex> u = User.new
%User{id: 0}
iex> User.encode u
""

I think sometimes a explicit User of id 0 is needed, exprotobuf does so:

iex> u = User.new id: 0
%User{id: 0}
iex> User.encode u
<<8, 0>>
iex> u = User.new
%User{id: nil}
iex> User.encode u
""

Good work!

Just wanted to say thanks. I was going to embark on something almost exactly like this after seeing how exprotobuf was implemented. Even without a protoc generator, I'll happily manually setup the DSL for the saner and much more efficient approach you're taking. Anxious for oneof to land :)

some variables unused warning (v0.6.1)

==> protobuf
Compiling 20 files (.ex)
warning: variable "name_atom" is unused (if the variable is not meant to be used, prefix it with an underscore)
lib/protobuf/dsl.ex:55

warning: variable "msg_props" is unused (if the variable is not meant to be used, prefix it with an underscore)
lib/protobuf/dsl.ex:297

warning: variable "type" is unused (if the variable is not meant to be used, prefix it with an underscore)
lib/protobuf/encoder.ex:188

warning: variable "type" is unused (if the variable is not meant to be used, prefix it with an underscore)
lib/protobuf/encoder.ex:189

warning: variable "v" is unused (if the variable is not meant to be used, prefix it with an underscore)
lib/protobuf/encoder.ex:189

Value that is too big for an uint32 is successfully encoded and decoded

I have this protocol buffer definition:

syntax = "proto3";

message Uint {
  uint32 u = 1;
}

When I create a value that's too big for a 32 bit unsigned integer, encoding and decoding surprisingly works:

iex(10)> Uint.decode(Uint.encode(Uint.new(u: 12345678901234567890)))
%Uint{u: 12345678901234567890}

If I use a different library to decode the result, I get a shorter value:

iex(9)> Uint.encode(Uint.new(u: 12345678901234567890))
<<8, 210, 149, 252, 216, 206, 177, 170, 170, 171, 1>>
iex(10)> Proto2Test.Uint.decode(<<8, 210, 149, 252, 216, 206, 177, 170, 170, 171, 1>>)
%Proto2Test.Uint{u: 3944680146}

I think the 3944680146 value looks better, definitely fits into a 32 bit unsigned integer.

proto3 with nil string should be valid

Validation fails when assigning nil to a string, but, as far as I can tell, this should be vali.

syntax = "proto3";

message Test {
  string value = 1;
}
...
# raises an error
Test.encode(Test.new(value: nil))

This works fine in Go. Also, encode should return an error, not raise. There should be an encode! for raising.

Obtain valid values and keys for enum

Assuming I have the following .proto file

message Foo {
  Bar bar = 1;
}
enum Bar {
  a = 0;
  b = 1;
}

Is there a way to obtain all the valid enum values/keys?

I know that I can switch between atom and integer with Bar.value(:a) and Bar.key(0), but I couldn't find a way to obtain something like Bar.keys() to obtain [:a, :b], Bar.values() to obtain [0, 1] or Bar.map() to obtain %{a: 0, b: 1}

My apologies if there's already a function implemented and I didn't find it. If there isn't, I think it could be helpful.

gen_descriptors and grpc seem to be mutually exclusive

Hi, forgive my ignorance of GRPC as I'm relatively new to the libraries but I've noticed what is perhaps a bug - gen_descriptors=true and plugins=grpc seem to be mutually exclusive.

$ protoc --elixir_out=gen_descriptors=true:./lib/ *.proto
$ protoc --elixir_out=gen_descriptors=true,plugins=grpc:./lib/ *.proto

What I mean is that if I specify gen_descriptors=true - the generated modules file doesn't contain any service or stub definitions.

Extensions

Cool, thanks @tony612 for this project. I feel this is the right direction for protobufs in Elixir.

Quick question: do you have thoughts on how we'd build extensions? I want to add something akin to:

service ExampleService {
  rpc ping (PingRequest) returns (PongResponse) {
  	option (google.api.http) = { post: "/ping" };
  }

  rpc status (StatusRequest) returns (StatusResponse) {
  	option (google.api.http) = { get: "/status" };
  }
}

This would allow us to use services like grpc gateway. I would expect this to build a service object similar to:

defmodule Defs.ExampleService do
  use Protobuf.Service

  rpc :ping, Defs.PingRequest, Defs.PongResponse, %{"google.api.http": %{post: "/ping"}}
  rpc :status, Defs.StatusRequest, Defs.StatusResponse, %{"google.api.http": %{post: "/status"}}
end

Also, would it be possible to make the service definitions generic and then the grpc library could easily build off of this. E.g., as above, we could just have the service definition be a simple module and the grpc could run something akin to:

in grpc library:

# ... somewhere
grpc.add_services([Defs.ExampleService])

# ... somewhere else
def add_services(services) do
  for service <- services do
    for rpc <- service.rpcs do
      add_route(rpc.name, rpc.request_message, rpc.response_message, rpc.options)
    end
  end
end

I've started looking into how to build extensions, but it's a bit mind-bending to track how to add that. Happy for your thoughts here.

Unable to handle recursive types

Hi,

Excellent library!

I'm looking at writing a client for Google Cloud Bigtable but ran into problems when using new on one of the types generated by protobuf-elixir.

The problem occurs because of the type RowFilter includes a property of type Condition which again include properties of type RowFilter. See this data.proto spec.

When calling new on RowFilter or RowFilter.Condition the __default_struct__ function is invoked recursively forever.

I havn't investigated a solution to the problem too much yet but wanted to bring it to you guys attention as your obviously much better informed on what a solution might be.

Again, thanks for all the great work on protobuf-elixir and grpc

Best regards

Simon, Denmark

No longer can decode protobuf encoded with version 0.5.0

Upgraded from 0.5.4 and receive the following errors when decoding a previously encoded protobuf.

The following arguments were given to Protobuf.Decoder.raw_decode_varint/3:                                                                                                                                                                                                                                                                            
                                                                                                                                                                                                                                                                                                                                                            
         # 1                                                                                                                                                                                                                                                                                                                                                
         ""                                                                                                                                                                                                                                                                                                                                                 
                                                                                                                                                                                                                                                                                                                                                            
         # 2                                                                                                                                                                                                                                                                                                                                                
         [0, 0, "500fb45b-6f99-486d-b976-acaeec980bf8", 2, 28, <<10, 36, 98, 101, 98, 53, 54, 53, 101, 50, 45, 57, 55, 52, 55, 45, 52, 51, 57, 51, 45, 56, 53, 102, 98, 45, 52, 48, 99, 51, 52, 53, 100, 52, 99, 98, 49, 102, 18, 32, 49, 99, 101, 102, ...>>, 2, 27, <<10, 36, 54, 98, 50, 101, 98, 49, 49, 101, 45, 97, 97, 98, 97, 45, 52, 50, 54, 98, 45
, 56, 50, 50, 100, 45, 53, 57, 49, 49, 51, 49, 102, 56, 97, 101, 55, 101, 18, 32, 57, ...>>, 2, 27, "adadda62-40cf-435f-86f9-49059633cf59", 2, 26, 1561137123509, 0, 25, 1561137123509, 0, 24, <<10, 16, 105, 115, 95, 99, 104, 105, 101, 102, 95, 106, 117, 115, 116, 105, 99, 101, 18, 4, 116, 114, 117, 101, 26, 7, 66, 79, 79, ...>>, 2, 23, "Women", 2,
 22, <<10, 16, 83, 79, 77, 69, 95, 72, 73, 71, 72, 95, 83, 67, 72, 79, 79, 76>>, 2, 21, <<10, 36, 97, 99, 97, 49, 56, 53, 48, 98, 45, 101, 54, 50, 48, 45, 52, 51, 53, 98, ...>>, 2, 20, <<10, 24, 74, 111, 104, 110, 32, 77, 97, 114, 115, 104, 97, 108, 32, 66, 105, ...>>, 2, 19, <<10, 21, 99, 104, 105, 101, 102, 95, 106, 117, 115, 116, 105, 99, ...>
>, 2, 18, "\b\"", 2, 17, "en-gb", 2, 16, "en-us", 2, 15, "Deist", 2, 14, ...]                                                                                                                                                                                                                                                                               
                                                                                                                                                                                                                                                                                                                                                            
         # 3                                                                                                                                                                                                                                                                                                                                                
         :value                                                                                                                                                                                                                                                                                                                                             
                                                                                                                                                                                                                                                                                                                                                            
     Attempted function clauses (showing 10 out of 10):                                                                                                                                                                                                                                                                                                     
                                                                                                                                                                                                                                                                                                                                                            
         defp raw_decode_varint(<<0::integer()-size(1), x::integer()-size(7), rest::bitstring()>>, result, type)                                                                                                                                                                                                                                            
         defp raw_decode_varint(<<1::integer()-size(1), x0::integer()-size(7), 0::integer()-size(1), x1::integer()-size(7), rest::bitstring()>>, result, type)                                                                                                                                                                                              
         defp raw_decode_varint(<<1::integer()-size(1), x0::integer()-size(7), 1::integer()-size(1), x1::integer()-size(7), 0::integer()-size(1), x2::integer()-size(7), rest::bitstring()>>, result, type)                                                                                                                                                 
         defp raw_decode_varint(<<1::integer()-size(1), x0::integer()-size(7), 1::integer()-size(1), x1::integer()-size(7), 1::integer()-size(1), x2::integer()-size(7), 0::integer()-size(1), x3::integer()-size(7), rest::bitstring()>>, result, type)                                                                                                    
         defp raw_decode_varint(<<1::integer()-size(1), x0::integer()-size(7), 1::integer()-size(1), x1::integer()-size(7), 1::integer()-size(1), x2::integer()-size(7), 1::integer()-size(1), x3::integer()-size(7), 0::integer()-size(1), x4::integer()-size(7), rest::bitstring()>>, result, type)                                                       
         defp raw_decode_varint(<<1::integer()-size(1), x0::integer()-size(7), 1::integer()-size(1), x1::integer()-size(7), 1::integer()-size(1), x2::integer()-size(7), 1::integer()-size(1), x3::integer()-size(7), 1::integer()-size(1), x4::integer()-size(7), 0::integer()-size(1), x5::integer()-size(7), rest::bitstring()>>, result, type)          
         defp raw_decode_varint(<<1::integer()-size(1), x0::integer()-size(7), 1::integer()-size(1), x1::integer()-size(7), 1::integer()-size(1), x2::integer()-size(7), 1::integer()-size(1), x3::integer()-size(7), 1::integer()-size(1), x4::integer()-size(7), 1::integer()-size(1), x5::integer()-size(7), 0::integer()-size(1), x6::integer()-size(7),
 rest::bitstring()>>, result, type)                                                                                                                                                                                                                                                                                                                         
         defp raw_decode_varint(<<1::integer()-size(1), x0::integer()-size(7), 1::integer()-size(1), x1::integer()-size(7), 1::integer()-size(1), x2::integer()-size(7), 1::integer()-size(1), x3::integer()-size(7), 1::integer()-size(1), x4::integer()-size(7), 1::integer()-size(1), x5::integer()-size(7), 1::integer()-size(1), x6::integer()-size(7),
 0::integer()-size(1), x7::integer()-size(7), rest::bitstring()>>, result, type)                                                                                                                                                                                                                                                                            
         defp raw_decode_varint(<<1::integer()-size(1), x0::integer()-size(7), 1::integer()-size(1), x1::integer()-size(7), 1::integer()-size(1), x2::integer()-size(7), 1::integer()-size(1), x3::integer()-size(7), 1::integer()-size(1), x4::integer()-size(7), 1::integer()-size(1), x5::integer()-size(7), 1::integer()-size(1), x6::integer()-size(7),
 1::integer()-size(1), x7::integer()-size(7), 0::integer()-size(1), x8::integer()-size(7), rest::bitstring()>>, result, type)                                                                                                                                                                                                                               
         defp raw_decode_varint(<<1::integer()-size(1), x0::integer()-size(7), 1::integer()-size(1), x1::integer()-size(7), 1::integer()-size(1), x2::integer()-size(7), 1::integer()-size(1), x3::integer()-size(7), 1::integer()-size(1), x4::integer()-size(7), 1::integer()-size(1), x5::integer()-size(7), 1::integer()-size(1), x6::integer()-size(7),
 1::integer()-size(1), x7::integer()-size(7), 1::integer()-size(1), x8::integer()-size(7), 0::integer()-size(1), x9::integer()-size(7), rest::bitstring()>>, result, type)                                                                                                                                                                                  
                                                                                                                                                                                                                                                                                                                                                            
     code: {:ok, result} = KVRepo.get_profile(p.id, [])                                                                                                                                                                                                                                                                                                     
     stacktrace:                                                                                                                                                                                                                                                                                                                                            
       (protobuf) lib/protobuf/decoder.ex:136: Protobuf.Decoder.raw_decode_varint/3                                                                                                                                                                                                                                                                         
       (protobuf) lib/protobuf/decoder.ex:13: Protobuf.Decoder.decode/2                                                                                                                                                                                                                                                                                     
       (profiles) lib/profiles/projections/protobuf.ex:79: Profiles.Projections.Protobuf.decode_profile_projection/1                                                                                                                                                                                                                                        
       (profiles) lib/profiles/repository/kv_repo/redis.ex:117: Profiles.Repository.KVRepo.Redis.get_profile/2                                                                                                                                                                                                                                              
       test/profiles/repository/kv_repo/redis_test.exs:156: (test)

I can send you the protocol buffer file if you need it.

Elixir terms converter extensions

Hi 👋

I would like to know if the maintainers are interested in a feature like this.

Let's say I have this message:

package MyApp.Proto;

message NaiveDateTime {
  uint32 microseconds = 1;
}

Then, when I working I'm my app, I've always to convert to an Elixir term.

message.birthday
%MyApp.Proto.NaiveDateTime{microseconds: ...}

# oh no, I have to remember to:
to_naive_date_time(message.birthday)
~N[1988-10-29 00:00:00]

I was wondering, wouldn't it be nice to have a way to tell the encoder/decoder how they work with some messages. Then, I could write in my app:

message.birthday
~N[2000-01-01 00:00:00]

I really would like to help to add this feature, but I would like to hear from you folks if you are interested.

Then, we can discuss the API and how this extension should work.

I had a suggestion, we could have something like:

message NaiveDateTime {
  option (elixirpb.message).converter = "MyApp.NaiveDateTimeConverter";
  uint32 microseconds = 1;
}

Then in my app, I implement some module behavior:

defmodule MyApp.NaiveDateTimeConverter do
  @behaviour Protobuf.Converter

  # or post_decode, decode, after_decode
  def to_elixir(%MyApp.Protobuff.NaiveDateTime{microseconds: microseconds}),
    do: NaiveDateTime.from_unix!(microseconds, :microsecond)

  # or pre_encode, encode, before_encode
  def to_protobuf(%DateTime{} = datetime) do
    microseconds = DateTime.to_unix(datetime, :microsecond)
    %MyApp.Protobuff.NaiveDateTime{microseconds: microseconds}
  end

  def to_protobuf(%NaiveDateTime{} = naive) do
    {:ok, datetime} = DateTime.from_naive(naive, "Etc/UTC")
    microseconds = DateTime.to_unix(datetime, :microsecond)
    %MyApp.Protobuff.NaiveDateTime{microseconds: microseconds}
  end

  def elixir_type, do: "NaiveDateTime.t() | DateTime.t()"
end

What do you folks think? Do we have space in this library for such a feature?

Thoughts behind the project

Hi @tony612, could you share the ideias you had/have behind this other implementation of protobufs to Elixir?

I'm currently working in a fork to simplify the way Exprotobuf deals with :gpb under the hood and now I'm wondering which are your motivations for this new project.

Cheers!

Error on Google.Protobuf.Struct.encode

I am receiving error when try to encode Google.Protobuf.Struct.
Example of executed code:
Google.Protobuf.Struct.encode(Google.Protobuf.Struct.new(fields: %{"valid" => {:bool_value, true}}))
Stacktrace:
** (Protobuf.EncodeError) Got error when encoding Google.Protobuf.Struct#fields: %Protobuf.EncodeError{message: "Got error when encoding Google.Protobuf.Struct.FieldsEntry#value: %Protocol.UndefinedError{description: \"\", protocol: Enumerable, value: {:bool_value, true}}"} (elixir) /home/build/elixir/lib/elixir/lib/enum.ex:1: Enumerable.impl_for!/1 (elixir) /home/build/elixir/lib/elixir/lib/enum.ex:141: Enumerable.reduce/3 (elixir) lib/enum.ex:3015: Enum.reduce/3 (protobuf) lib/protobuf/builder.ex:12: Protobuf.Builder.new/2 (protobuf) deps/protobuf/lib/protobuf/encoder.ex:14: Protobuf.Encoder.encode/3 (protobuf) deps/protobuf/lib/protobuf/encoder.ex:83: anonymous fn/5 in Protobuf.Encoder.encode_field/3 (protobuf) deps/protobuf/lib/protobuf/encoder.ex:50: anonymous fn/5 in Protobuf.Encoder.encode!/2 (elixir) lib/enum.ex:1940: Enum."-reduce/3-lists^foldl/2-0-"/3 (protobuf) deps/protobuf/lib/protobuf/encoder.ex:20: Protobuf.Encoder.encode/2 (protobuf) deps/protobuf/lib/protobuf/encoder.ex:83: anonymous fn/5 in Protobuf.Encoder.encode_field/3 (elixir) lib/enum.ex:1331: anonymous fn/3 in Enum.map/2 (stdlib) maps.erl:232: :maps.fold_1/3 (elixir) lib/enum.ex:1956: Enum.map/2 (protobuf) deps/protobuf/lib/protobuf/encoder.ex:50: anonymous fn/5 in Protobuf.Encoder.encode!/2 (elixir) lib/enum.ex:1940: Enum."-reduce/3-lists^foldl/2-0-"/3 (protobuf) deps/protobuf/lib/protobuf/encoder.ex:20: Protobuf.Encoder.encode/

I think that is because nil passed as argument to Protobuf.Encoder.is_enum_default/2 and is_atom(v) guard thinks that it is valid.
I can fix it by adding not is_nil check to function, but I don't know if it is a right way to solve this problem.

TravisCI + Ebert

Setup all the automated checks

Feature Request: support google.rpc.Status.Details

The gRPC error model has the ability to embed any kind of message in the status of a RPC (see this doc).

It is a feature that would be quite useful to convey error details since custom status codes are not documented in .proto files.

Support Google Protobuf Timestamp type

First of all, I must thank you for this great GRPC lib for Elixir. The features are almost there.

However, one of well known type is still missing, Timestamp. I'd already implement the Google.Protobuf.Timestamp type and would like to contribute back but not sure where I should put this chunk of code into.

defmodule Google.Protobuf.Timestamp do
  @moduledoc false

  use Protobuf, syntax: :proto2

  @type t :: %__MODULE__{
    seconds: integer,
    nanos: integer
  }

  defstruct [:seconds, :nanos]

  field :seconds, 1, optional: true, type: :int64
  field :nanos, 2, optional: true, type: :int32
end

This implement Timestamp type according to Google Protobuf Timestamp type message definition here

If you agree with this implementation, I will open PR as soon as possible.

Integration with mix

I've been playing around with trying to make protobuf compilation happen during a project's mix compile, and it seems to be pretty feasible. Would this be an appropriate feature for this repo? Would you be up to review a PR to implement this?

Function clause exception when decoding garbage binaries

Hi,

When I try to decode an obviously wrong binary as a protobuf struct I get a function clause exception, e.g.:

iex(2)> Protobuf.Decoder.decode(<<1,1,1>>, SomeProtoStruct)
** (FunctionClauseError) no function clause matching in Protobuf.Decoder.raw_decode_value/3    
    
    The following arguments were given to Protobuf.Decoder.raw_decode_value/3:
    
        # 1
        1
    
        # 2
        <<1, 1>>
    
        # 3
        [1, 0]
    
    Attempted function clauses (showing 4 out of 4):
    
        def raw_decode_value(0, <<bin::bitstring()>>, result)
        def raw_decode_value(2, <<bin::bitstring()>>, result)
        def raw_decode_value(5, <<n::integer()-size(32), rest::bitstring()>>, result)
        def raw_decode_value(1, <<n::integer()-size(64), rest::bitstring()>>, result)
    
    (protobuf) lib/protobuf/decoder.ex:283: Protobuf.Decoder.raw_decode_value/3
    (protobuf) lib/protobuf/decoder.ex:13: Protobuf.Decoder.decode/2

The thing is, I'd like to be able to distinguish these errors from other function clause exceptions that might happen in the same process. Currently the only way is to wrap the decode like this:

 defp decode(msg) do
    SomeProtoStruct.decode(msg)
  catch
    _, _ ->
      raise Protobuf.DecodeError
  end

Could the auto-generated decode function not do this? I'm willing to make a PR for this if you agree.

Error when parsing packages without an explicit package name

The following proto file, fails to compile for me:

syntax = "proto3";

message Test {
    string tst = 1;
}

It returns the following error:

** (FunctionClauseError) no function clause matching in String.split/3

    The following arguments were given to String.split/3:

        # 1
        nil

        # 2
        "."

        # 3
        []

    (elixir) lib/string.ex:383: String.split/3
    lib/protobuf/protoc/generator/util.ex:33: Protobuf.Protoc.Generator.Util.normalize_pkg_name/1
    lib/protobuf/protoc/generator/util.ex:11: Protobuf.Protoc.Generator.Util.attach_pkg/2
    lib/protobuf/protoc/generator/message.ex:22: Protobuf.Protoc.Generator.Message.parse_desc/2
    lib/protobuf/protoc/generator/message.ex:11: Protobuf.Protoc.Generator.Message.generate/2
    (elixir) lib/enum.ex:1294: Enum."-map/2-lists^map/1-0-"/2
    lib/protobuf/protoc/generator.ex:24: Protobuf.Protoc.Generator.generate_content/2
    lib/protobuf/protoc/generator.ex:11: Protobuf.Protoc.Generator.generate/2
--elixir_out: protoc-gen-elixir: Plugin failed with status code 1.

Adding package test; to the proto file fixes the problem.
However, since the --ccp_out generator seems to accept the original proto file, I don't think this behaviour is intended.

I don't have a lot of experience with protocol buffers, so forgive me if I'm doing something incredibly wrong here :).

Multiple conflicting PbExtension modules

protobuf-elixir: 0.8.0-beta.1
elixir: 1.10.1
otp: 22.2.7

I am trying to compile these .proto files using version 0.8.0-beta.1, and in most of the resulting 155 elixir files I get a PbExtension module. Elixir can obviously only handle one module with a given name, so this fails to compile.

v0.7.1 erroneously enforces source code ordering of modules

Just upgraded from 0.6.3 to 0.7.1.

On 0.6.3, I could have generated Elixir protobuf files like this:

defmodule Test.MyMessage do
  @moduledoc false
  use Protobuf, syntax: :proto3

  @type t :: %__MODULE__{
          type: atom | integer
        }
  defstruct [
    :type
  ]
  field :type, 1, type: Test.MessageEnum, enum: true
end

defmodule Test.MessageEnum do
  @moduledoc false
  use Protobuf, enum: true, syntax: :proto3

  field :ZERO, 0
  field :ONE, 1
  field :TWO, 2
end

Compiling the app with 0.7.1 gives errors like these when compiling the above generated protobuf files:

== Compilation error in file /.../file.pb.ex ==
** (UndefinedFunctionError) function Test.MessageEnum.key/1 is undefined (module Test.MessageEnum is not available)
    Test.MessageEnum.key(0)
    lib/protobuf/dsl.ex:33: anonymous fn/2 in Protobuf.DSL."MACRO-__before_compile__"/2
    (elixir) lib/enum.ex:1948: Enum."-reduce/3-lists^foldl/2-0-"/3
    expanding macro: Protobuf.DSL.__before_compile__/1
    /.../file.pb.ex:1: Test.MyMessage (module)

Going back to 0.6.3 fixes the issue, as does re-ordering the source code like so:

defmodule Test.MessageEnum do
  @moduledoc false
  use Protobuf, enum: true, syntax: :proto3

  field :ZERO, 0
  field :ONE, 1
  field :TWO, 2
end

defmodule Test.MyMessage do
  @moduledoc false
  use Protobuf, syntax: :proto3

  @type t :: %__MODULE__{
          type: atom | integer
        }
  defstruct [
    :type
  ]
  field :type, 1, type: Test.MessageEnum, enum: true
end

Given this code is generated, re-ordering it should not be necessary.

On glancing at the recent changes, I'd guess that the problem is coming from here: cfb026b#diff-8b2c39dee263c97e81c3eb547e5012d8

This appears to be a bug to me since Elixir code shouldn't care about the ordering of modules within source code during compilation. Thanks!

Protobuf module conflicts with exprotobuf

The top level module name Protobuf is also the top level module for exproto (https://github.com/bitwalker/exprotobuf/blob/master/lib/exprotobuf.ex). This makes it difficult to use protobuf-elixir in projects that also have a secondary dependency on exproto. I know it's a lot to ask to rename it but I wanted to bring it up here. It's likely that others have or will run into this also.

In my case I'm unable to use https://github.com/peburrows/diplomat because of it's dependency on exproto.

[question] Generate proto files according to package

Is there a way to generate the proto files into a directory structure matching the proto package ?

For instance I have a proto definition placed in priv/proto such as:

syntax = "proto3";
package company.my_messages;
...

When I compile the messages using protoc --elixir_out=./lib priv/proto/*.proto I'd like the files to be generated in lib/company/my_message.

Installing the proto plugin

Is it supposed to be
mix escript.install hex protobuf or mix archive.install hex protobuf
Because Phoenix uses the latter.

Support for Value Wrappers

In protobuf messages when we have enum or scalar types they can't be nil. To make it nil, a way of doing it is to wrap in a message. This pattern is very common, we can see for example in google protobufs wrappers:

https://github.com/protocolbuffers/protobuf/blob/master/src/google/protobuf/wrappers.proto

And it makes writing code like:

message User {
  StringValue first_name = 1;
}

Later in Elixir, we'll have to access like:

# when present
user.first_name
%Google.Protobuf.StringValue{value: "Abilidebob"}

# when not present
user.first_name
%Google.Protobuf.StringValue{value: nil}

I wonder if it would not be cool this library can understand this pattern and unwrap and wrap automatically for us when decoding/encoding?

Then we could have something like:

# when present
user.first_name
"Abilidebob"

# when not present
user.first_name
nil

Of course, to make it compatible with previous versions, it should by default false, but you can turn as an option.

For example:

protoc --elixir_out=./lib --plugin=./protoc-gen-elixir --elixir_opt=value_wrapper=true myproto.proto

What do you folks think? Do we have space for such a feature in this library?

one question for encoder?

https://github.com/tony612/protobuf-elixir/blob/5f8da613191501d560dce00087cb500f654a991a/lib/protobuf/encoder.ex#L54

For example, I need encode

[ [], [] ]

just like :

syntax = "proto3";

message TListList {
    repeated TList listlist = 1;
}

message TList {
    repeated string ele = 1;
}

But, after encode I got

// ==>
// ""

and after decode it, I got

// ==>
// [ ]

It's not the result I need.

Feature request: Optionally populate sub-messages

When a message has fields which are in turn messages it would be handy to optionally have .new create the sub-messages all the way through the hierarchy defined in the message structure.

gRPC rpc call should be snake_case in Elixir

First of all, thanks for your work! It is amazing.

I have one issue though:
According to Protocol Buffers style guide https://developers.google.com/protocol-buffers/docs/style#services rpc calls in Service section should CamelCased:
rpc SayHello(HelloRequest) returns (HelloResponse);
which is then translated to following Elixir code:
rpc :SayHello, HelloRequest, HelloResponse
and we can use it in the server as:

defmodule HelloExample.Server do
  use GRPC.Server, service: HelloExample.Service

  def SayHello(request, stream) do
    ...
  end
end

but Elixir functions should be snake_cased (see https://hexdocs.pm/elixir/master/naming-conventions.html#casing ).

If I rename .proto definition to snake_case protoc-gen-lint is complaining if I rename it to CamelCase Elixir is complaining.

Is it possible to convert the CamelCased proto definition to snake_case Elixir definition so both linters are satisfied?

protoc --elixir_out=./lib helloworld.proto

可以介绍一下在Windows系统上怎么安装这个吗？我尝试用macOS的教程，运行到：
proto>protoc --elixir_out=./lib helloeworld.proto 时候总是提示我：--elixir_out: protoc-gen-elixir: 系统找不到指定的文件。但实际上在上一步我是生成了这个文件的。

mix escript.install hex protobuf
Failed to fetch record for 'hexpm/protobuf' from registry (using cache)
:timeout
Resolving Hex dependencies...
Dependency resolution completed:
New:
�[32m protobuf 0.5.4�[0m

Getting protobuf (Hex package)
Request failed (:timeout)
If this happens consistently, adjust your concurrency and timeout settings:

HEX_HTTP_CONCURRENCY=1 HEX_HTTP_TIMEOUT=120 mix deps.get
Fetch failed. Using locally cached package (c:/Users/Paul.Qing/.hex/packages/hexpm/protobuf-0.5.4.tar)
All dependencies up to date
Compiling 21 files (.ex)
Generated protobuf app
Generated escript protoc-gen-elixir with MIX_ENV=prod
Found existing entry: c:/Users/Paul.Qing/.mix/escripts/protoc-gen-elixir。

希望收到回复，谢谢

Feature Request: Encode to / decode from JSON

In proto3 google added an official JSON mapping (https://developers.google.com/protocol-buffers/docs/proto3#json). Is there any plan to support encoding to and decoding from JSON?

[question] Generate proto files automatically

Is there an easy way to generate the proto files automatically when calling mix compile or something? So I don't have to manually run the protoc command manually.

Oneof value is lost when set to type default value

Protobuf:

syntax = "proto3";

package example;

message Foo {
  oneof bar {
    int64 int = 1;
    string string = 2;
  }
}

Example of invalid behavior:

iex(1)> Example.Foo.new(bar: {:string, "x"}) |> Example.Foo.encode |> Example.Foo.decode
%Example.Foo{bar: {:string, "x"}} # WORKING AS EXPECTED

iex(2)> Example.Foo.new(bar: {:string, ""}) |> Example.Foo.encode |> Example.Foo.decode
%Example.Foo{bar: nil} # VALUE LOST

iex(3)> Example.Foo.new(bar: {:int, 1}) |> Example.Foo.encode |> Example.Foo.decode
%Example.Foo{bar: {:int, 1}} # WORKING AS EXPECTED

iex(4)> Example.Foo.new(bar: {:int, 0}) |> Example.Foo.encode |> Example.Foo.decode     
%Example.Foo{bar: nil} # VALUE LOST

Additional information:
Issue was not present prior 4a2d61f

DecodeError on windows

syntax = "proto3";
//012345678901
message Test
{
}

elixir --version
Erlang/OTP 20 [erts-9.3] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:10]
Elixir 1.7.2 (compiled with Erlang/OTP 19)

protoc.exe --version
libprotoc 3.6.1

%Google.Protobuf.SourceCodeInfo.Location{leading_comments: "0123456789", leading_detached_comments: [], path: [0, 4], span: [1, 4, 0, 2], trailing_comments: nil}
48 6 0
** (Protobuf.DecodeError) Google.Protobuf.SourceCodeInfo.Location: wrong wire_type for leading_detached_comments: got 0, want 2
    lib/protobuf/decoder.ex:36: Protobuf.Decoder.do_decode/3
    lib/protobuf/decoder.ex:74: Protobuf.Decoder.put_field/5
    lib/protobuf/decoder.ex:31: Protobuf.Decoder.do_decode/3
    lib/protobuf/decoder.ex:74: Protobuf.Decoder.put_field/5
    lib/protobuf/decoder.ex:31: Protobuf.Decoder.do_decode/3
    lib/protobuf/decoder.ex:74: Protobuf.Decoder.put_field/5
    lib/protobuf/decoder.ex:31: Protobuf.Decoder.do_decode/3
    lib/protobuf/protoc/cli.ex:8: Protobuf.Protoc.CLI.main/1
--elixir_out: protoc-gen-elixir: Plugin failed with status code 1.

Merging cowboy/cowlib changes upstream ?

Are there any plans to merge the cowboy/cowlib forks upstream? I'm thinking the forks might miss out on security fixes/new functionality over time.

Feature Request: Performance

Hello.

Was wondering if there's any interest in making changes to improve performance? I just started looking at this and am taking it step by step. I'm looking at encoding first, and a simple benchmark compared to Jason, for one of our objects (nothing too special about it), doesn't look great:

Jason.encode         100000   25.63 µs/op
Protobuf.encode       50000   69.15 µs/op

If we remove the call to Protobuf.Validator.validate!(struct) in Encoder.encode (which is, by far, the lowest hanging fruit), we get that down to 33.31 µs/op.

I was thinking of two options (possibly adding support for both)

1 - Support a [validate: false] option to encode/2. This is trickier than it seems since opts isn't currently passed to child structure (although, it feels like it would be a good idea to do so anyways)

2 - Support a global config which enabled/disables validation

We'd probably use 2, enabling validation in dev/test, but disabling it in prod.

I'm happy to work on a PR for this, unless a) you don't want it or b) rather implement it yourself.

Ultimately, I think i'd be great (and reasonable) to get better performance than any JSON encoder.

protoc package name and path

Is there a reason package names are underscored?
package Foo.Bar --> defmodule Foo_Bar, why?
Can it listen to --elixir_out and not append a proto sub-folder?

Raise exceptions if incorrect types are passed into `new`

Currently, no checks are done when passing attributes to new/2. This means that it's possible to pass incorrect types or sub-messages while constructing a new struct. In most cases these errors won't be caught until the message is decoded since the encoder doesn't perform validations while it's encoding. To make this more concrete let's say that we have these definitions:

defmodule Foo do
  @moduledoc false
  use Protobuf, syntax: :proto3

  defstruct [:id]

  field :id, 1, type: :int32
end

defmodule Bar do
  @moduledoc false
  use Protobuf, syntax: :proto3

  defstruct [:id]

  field :id, 1, type: :string
end

defmodule Baz do
  @moduledoc false
  use Protobuf, syntax: :proto3

  defstruct [:foo]

  field :foo, 1, type: Foo
end

It's currently possible to do this:

Baz.new(foo: Bar.new(id: "test")) |> Baz.encode()
<<10, 6, 10, 4, 116, 101, 115, 116>>

Errors like this won't be caught until the client attempts to decode the message on the other end.

Proposal

I'd like to suggest that we check inputs as part of new and raise an exception if the value doesn't match the correct type. I believe that raising is the correct approach here because calling new with invalid data is equivalent to an ArgumentError. I don't believe that there's any recovery the user could take if they pass the wrong arguments to the function and its better to raise loudly so they're aware of the issue.

Obviously this would only work if the user calls new and doesn't work if they build the struct themselves. I'm not sure if that's an acceptable tradeoff or not.

I'm happy to work on submitting a PR for these changes if you believe that the idea has merit. But I wanted to make sure you thought this was a reasonable addition before I did the work on it.

How to get atom as value of enum field

If I define message:

message ExitStatus {
  enum Status {
    PASS = 0;
    FAIL = 1;
  }

  Status status = 1;
}

And later in the code

...
v = ExitStatus.decode(...)

Is there a way to get :PASS or :FAIL out of v?

Thank you.

float and NaN/Infinity values

Hello,

I found that it is not possible to work with NaN, Infinity, -Infinity values for float. Erlang doesn't have such values but they can be encoded in Protobuf messages. Currently, decoding of messages with such values fails with MatchError exception. Do you have some plans to support them?