The prost crate plugin is not working might need a new tag pushed?

  - remote: buf.build/prost/plugins/crate:v0.1.5-2
    out: .
    opt:
      - include_file=src/gen/mod.rs
      - gen_crate

Failure: plugin "buf.build/prost/plugins/crate:v0.1.5-2" failed: No such file or directory (os error 2)

Hi, I'm trying to generate code for the buf schema registry protobufs.

It generates the code but getting 2 compiler errors:

error[E0433]: failed to resolve: could not find `compiler` in `pbjson_types`
   --> src/gen/buf.alpha.registry.v1alpha1.rs:555:61
    |
555 |     pub responses: ::prost::alloc::vec::Vec<::pbjson_types::compiler::CodeGeneratorResponse>,
    |                                                             ^^^^^^^^ could not find `compiler` in `pbjson_types`

error[E0283]: type annotations needed
   --> src/gen/buf.alpha.registry.v1alpha1.rs:550:28
    |
550 | #[derive(Clone, PartialEq, ::prost::Message)]
    |                            ^^^^^^^^^^^^^^^^ cannot infer type
    |
    = note: cannot satisfy `_: std::default::Default`
    = note: this error originates in the derive macro `::prost::Message` (in Nightly builds, run with -Z macro-backtrace for more info)

The generated code for this section looks like this:

#[derive(Clone, PartialEq, ::prost::Message)]
pub struct GeneratePluginsResponse {
    /// Contains all the responses from the generated plugins. The order
    /// is defined by the order of the plugins in the request.
    #[prost(message, repeated, tag="1")]
    pub responses: ::prost::alloc::vec::Vec<::pbjson_types::compiler::CodeGeneratorResponse>,
    /// An optional array defining runtime libraries that the generated code
    /// requires to run, as specified by the plugin author. This may contain
    /// duplicate entries as the generation can be the result of multiple plugins,
    /// each of which declares its own runtime library dependencies. The libraries
    /// returned are lexicographically ordered by their name, but not deduplicated.
    /// How to handle duplicate libraries is left to the user.
    #[prost(message, repeated, tag="2")]
    pub runtime_libraries: ::prost::alloc::vec::Vec<RuntimeLibrary>,
}

My buf.gen.yaml:

version: v1
plugins:
  - remote: buf.build/prost/plugins/prost:v0.2.1-1
    out: src/gen
    opt:
      - bytes=.
      - compile_well_known_types
      - extern_path=.google.protobuf=::pbjson_types
      - file_descriptor_set
  - remote: buf.build/prost/plugins/serde:v0.2.1-1
    out: src/gen
  - remote: buf.build/prost/plugins/tonic:v0.2.1-1
    out: src/gen
    opt:
      - compile_well_known_types
      - extern_path=.google.protobuf=::pbjson_types
  - remote: buf.build/prost/plugins/crate:v0.3.1-1
    out: src/gen
    opt:
      - no_features

And running it via buf generate buf.build/bufbuild/buf:68da9dcf7ad819b7725e798deaf9ced5fe782e0f

My cargo.toml dependencies:

bytes = "1.1.0"
prost = "0.11.0"
pbjson = "0.5"
pbjson-types = "0.5.1"
serde = "1.0"
tonic = { version = "0.8", features = ["gzip"] }

I don't know if this is a bug or if I'm doing something wrong?

It's great to see this plugin for polyglot environments that compile protos directly. I don't use buf so currently it seems tricky to use the compiler (need to install rust and then build the crate). If precompiled binaries could be published to the github releases for popular platforms (maybe darwin/linux/windows amd64/arm64), that would make it much simpler to use this plugin with protoc.

I see that it's supported within protoc-gen-prost-crate - though I don't specifically want to generate a crate for the types and instead just want to generate a set of files for me to consume within my existing crate.

I'd love to have it as part of the base codegen package.

So we just recently ran into an issue with the generated Rust code from the ProtoBuf not serializing correctly. Specifically it was converting integers into strings for some reason. When I went digging I saw that the generated serde implementation was incorrectly converting integers to strings. This seemed to only happen for i64 as our u32 fields were correctly serialized.

message Amount {
  // ISO-4217 currency code to denote remitting promissory files in a specific
  // currency
  string currency_code = 1;
  // Signed 64 bit representation of the significand
  int64 amount         = 2;
  // Unsigned 32 bit representation of the decimal precision, i.e. the inverse
  // of the exponent. The allowed value is determined by the rust_decimal crate,
  // which allows a scale (decimal) between 0-28 inclusive.
  // e.g. amount = 10, decimals = 2 => value 0.01
  uint32 decimals      = 3;
}

#[derive(Clone, PartialEq, ::prost::Message)]
pub struct Amount {
    ///  ISO-4217 currency code to denote remitting promissory files in a specific
    ///  currency
    #[prost(string, tag="1")]
    pub currency_code: ::prost::alloc::string::String,
    ///  Signed 64 bit representation of the significand
    #[prost(int64, tag="2")]
    pub amount: i64,
    ///  Unsigned 32 bit representation of the decimal precision, i.e. the inverse
    ///  of the exponent. The allowed value is determined by the rust_decimal crate,
    ///  which allows a scale (decimal) between 0-28 inclusive.
    ///  e.g. amount = 10, decimals = 2 => value 0.01
    #[prost(uint32, tag="3")]
    pub decimals: u32,
}

But the serializer implementation handles the amount property like this:

        if self.amount != 0 {
            struct_ser.serialize_field("amount", ToString::to_string(&self.amount).as_str())?;
        }

Which converts the type from a i64 to a String. Its important to note that it doesn't seem to have a similar problem with u32:

        if self.decimals != 0 {
            struct_ser.serialize_field("decimals", &self.decimals)?;
        }

As the decimals are serialized properly.

I've been trying to use an external type in my proto file and it now finally works with protoc-gen-prost-crate but not protoc-gen-prost-serde.

The proto file:

import "google/rpc/status.proto";

message CommandResult {
    google.rpc.Status status = 1 ;
}

The buf.gen.yaml:

version: v1

plugins:
  - remote: buf.build/prost/plugins/prost:v0.2.0-2
    out: target/gen
    opt:
      - bytes=.
      - extern_path=.google=::google_api_proto::google
  - remote: buf.build/prost/plugins/serde:v0.2.0-2
    out: target/gen
    opt:
      - no_include
      - extern_path=.google=::google_api_proto::google

I've added google-api-proto = { version = "1.38.0", features = ["google-rpc"] } to Cargo.toml to provide the external type.

The error:

error[E0277]: the trait bound `Status: Serialize` is not satisfied
     |
331  |             struct_ser.serialize_field("status", v)?;
     |                        ---------------           ^ the trait `Serialize` is not implemented for `Status`

It seems that protoc-gen-prost-serde does not generate anything for the external type Status.

Is there a solution?

In the Buf plugin metadata, we include links to the license in each plugin. These links are not currently working for the latest releases because of missing tags:

• https://github.com/neoeinstein/protoc-gen-prost/blob/protoc-gen-prost-v0.2.3/LICENSE
• https://github.com/neoeinstein/protoc-gen-prost/blob/protoc-gen-tonic-v0.3.0/LICENSE

Are these published tags part of the normal release process?

Hi,

In Rust 1.63.0 there is a new clippy lint https://rust-lang.github.io/rust-clippy/master/index.html#derive_partial_eq_without_eq

This lint will complain about types that derive PartialEq and could implement Eq. See aforementioned link for details.

Currently, protoc-gen-prost will generate code that derives PartialEq but not Eq even though it could and that triggers this clippy lint.

Example:

error: you are deriving `PartialEq` and can implement `Eq`
 --> somefile.rs:3:17
  |
3 | #[derive(Clone, PartialEq, ::prost::Message)]
  |                 ^^^^^^^^^ help: consider deriving `Eq` as well: `PartialEq, Eq`
  |
  = note: `-D clippy::derive-partial-eq-without-eq` implied by `-D warnings`
  = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#derive_partial_eq_without_eq

It is sometimes common to treat clippy warnings as errors. At my company we do, so currently this is causing CI failure for us.

Is this issue arising from protoc-gen-prost? If not, please let me know where I should file this issue instead.

As per the docs:

A package named helloworld.abstract.v1 will have an output filename of helloworld.abstract.v1.rs.

While this is a nice readable format having a dot in the file name makes it incompatible with rust for use directly with mod. The result is we are required to use include macro. But IDE's such as vscode with rust extension are unable to reference the definition for traits etc. This is sort of a big deal for developer UX.

If we allowed the separator to be configurable IE _ helloworld_abstract_v1.rs. would be valid. That would allow the below which also allows visibility into the trait source via IDE.

mod helloworld_abstract_v1;
mod helloworld_abstract_v1_tonic;

#[cfg(feature = "helloworld")]
pub mod helloworld {
    pub use super::helloworld_abstract_vs::*
    pub use super::helloworld_abstract_vs_tonic::*
}

I imagine this would mess with your prost-crate assumptions so perhaps we could gate this an not compatible? Or better yet we could adopt this output? :)

When using buf to compile protos, no dependency are compiled to .rs files, and the rust build fails. For example, using anything from googleapis other than google.protobuf (as those are provided by prost-types) leads to broken rust code.

Issue description

# buf.gen.yaml
version: v1
plugins:
  - name: prost-crate
    strategy: all
    out: .
    opt: [gen_crate]
  - name: prost
    out: src

# buf.work.yaml
version: v1
directories:
  - .

# buf.yaml
version: v1
name: buf.build/foo/bar
deps:
  - buf.build/googleapis/googleapis

// hello.proto
syntax = "proto3";
package hello;

import "google/protobuf/timestamp.proto";
import "google/rpc/status.proto";

message Hello {
  google.rpc.Status status = 1;
  google.protobuf.Timestamp timestamp = 2;
}

# Cargo.toml
[package]
name = "hello"
version = "1.0.0"
authors = []
edition = "2021"

[dependencies]
prost = "0.10"
prost-types = "0.10"
tonic = { version = "0.7", features = ["compression"] }

[features]
# @@protoc_deletion_point(features)
# This section is automatically generated by protoc-gen-prost-crate.
# Changes in this area may be lost on regeneration.
## @@protoc_insertion_point(features)

With the above example, running buf mod update && buf generate runs successfully, but the resulting crate is broken:

$ tree src
src
├── hello.rs
└── lib.rs

the google.rpc.rs file is missing from the tree (protoc-gen-prost issue), and it is also missing from the include file (protoc-gen-prost-crate issue):

// @generated
#[cfg(feature = "hello")]
// @@protoc_insertion_point(attribute:hello)
pub mod hello {
    include!("hello.rs");
    // @@protoc_insertion_point(hello)
}

cargo log:

$ cargo check --features hello
    Checking hello v1.0.0 (/home/tuetuopay/dev/hello)
error[E0433]: failed to resolve: could not find `google` in the crate root
 --> src/hello.rs:5:47
  |
5 |     pub status: ::core::option::Option<super::google::rpc::Status>,
  |                                               ^^^^^^ could not find `google` in the crate root

error[E0283]: type annotations needed
 --> src/hello.rs:2:28
  |
2 | #[derive(Clone, PartialEq, ::prost::Message)]
  |                            ^^^^^^^^^^^^^^^^ cannot infer type
  |
  = note: cannot satisfy `_: Default`
  = note: this error originates in the derive macro `::prost::Message` (in Nightly builds, run with -Z macro-backtrace for more info)

Some errors have detailed explanations: E0283, E0433.
For more information about an error, try `rustc --explain E0283`.
error: could not compile `hello` due to 2 previous errors

Issue root cause

The cause comes either from buf or by the generators, I don't know which one is incorrect.

buf does not run plugins with .proto from dependencies in the file_to_generate field of CodeGeneratorRequest, and these plugins do filter the files to output only to files in this list (see protoc-gen-prost/src/lib.rs:86).

I tend to think this is an issue with the generator, as raw prost-build does not expose such behaviour, hence why the issue is raised here.

Hi!

When generate the serde value with pbjson, this emit_vfields option is very handy when we don't want to emit the default value.

Really looks forward that protoc-gen-prost can support this. Does this lib has concern of implementing this option? If all things are good I could file a PR.

• influxdata/pbjson#82 (comment)
• https://docs.rs/pbjson-build/latest/pbjson_build/struct.Builder.html#method.emit_fields

First of all, thank you so much for making these -- this is the right way to do protobuf generation across a bigger/multi-language world, as opposed to via Rust's build.rs!

I filed tokio-rs/console#349 over on tokio's console-api because I'm in sort of a catch-22. Because I generated my protos with protoc-gen-tonic, they have the "compression" feature enabled which emits, in the generation, fields that require my library consuming them to have "compression" enabled as well.

That's fine if it's just me, but because console-api has their own generated protos with compression disabled, I get trouble when I try to compile console-subscriber into my application because I've already got tonic and it enables "compression" which breaks their generated protos' expectations.

IMO, they should just have a more general proto committed. But I figured I'd send an issue along in case other folks see it, or to see if there's a convenient way to enable or disable the compression features from the generated tonic files.

Hi,

I have a failure compiling some protos with buf:

version: v1
plugins:
  - plugin: prost
    out: gen
    opt:
      - bytes=.
      - compile_well_known_types
      - extern_path=.google.protobuf=::pbjson_types
      - file_descriptor_set

Failure: plugin prost: envoy.service.auth.v2alpha.rs: does not exist

I'm using prost plugin compiled from main (44d3fc3).

The failure happens because the prost code generation might omit empty files but the file_descriptor_set option makes it needed. Removing file_descriptor_set makes it work.

The problem starts with commit 038cd34 when prost-build crate was upgraded.

I have a draft PR that fixes the issue, but I don't think it should be merged as is:

34c2ac3

Hi 👋

I wrote axum-connect, which implements the Connect protocol for use with axum. It's designed to mix proto-based services into existing codebases, happily co-existing alongside more traditional HTTP endpoints. Seeing as you target buf.build I'll digress and assume prior knowledge in this area.

The code generator for axum-connect leaves an awful lot to be desired. It doesn't fit well into polyglot codebases (which is frankly the whole point of axum-connect) nor does it work at all with buf.build. I've had a moonshot item on my list since inception: to support buf.build remote execution for code gen. I just came across your codebase today.

Would you be willing to accept a PR to add support for axum-connect service definitions? I think I can do most (maybe all) the heavy lifting. This would involve:

• A new directory/workspace-member protoc-gen-axum-connect
  • The code generator is already based on prost
  • There were a lot code-gen hacks I added that need to be done away with
  • Supports any axum-connect specific configuration. None as of right now
  • Determine if we should require protoc-gen-prost (or possibly protoc-gen-prost-crate) as well as protoc-gen-prost-serde. Both are required for axum-connect to make any logical sense. I don't know enough yet to say what the right answer is here; open to suggestions?
• Publish protoc-gen-axum-connect to buf.build's community plugins for remote code-gen

The code generator itself is very simple, it adds a single function (scoped to a mod) for each proto service RPC. That function is analogous to axum's get and post functions, that wrap a high-order function in a future for use with Axum's extractors. Overall axum-connect is a very simple project, the bulk of the complexity is in the monstrously complicated type definitions for handlers.

The easiest path to have my Rust modules mirror Protobuf packages is to have a directory structure like foo/bar/v1.rs, especially if I want to other files in the same modules (say, foo::bar::random for a random generation module).

To that end, it would make a lot of sense for me to have an option (for both protoc-gen-prost and protoc-gen-prost-serde) to only use the last package name in the generate Rust file name (essentially the basename of the proto file). This would probably mean that protoc-gen-prost needs to create an appropriate directory tree too.

Hi,

protoc-gen-prost-serde doesn't generate ${package}.serde.rs for packages with "type" in their name,
e.g: envoy.type.v3.rs, while it does for any other package.

version: v1
plugins:
  - remote: buf.build/prost/plugins/prost:v0.2.1-1
    out: gen
    opt:
      - bytes=.
      - compile_well_known_types
      - extern_path=.google.protobuf=::pbjson_types
      - file_descriptor_set
  - remote: buf.build/prost/plugins/serde:v0.2.1-1
      out: gen
      opt:
        - extern_path=.google.protobuf=::pbjson_types
  - remote: buf.build/prost/plugins/tonic:v0.2.1-1
    out: gen
    opt:
      - compile_well_known_types
      - extern_path=.google.protobuf=::pbjson_types
  - name: prost-crate
    out: .
    strategy: all
    opt:
      - gen_crate=Cargo.tpl.toml
      - include_file=gen/mod.rs
      - no_features

Hi!

I want to compile my generated rust protos to wasm-unknown-unknown but the generate ::connect() method doesn't compile to wasm. The connect method can be omitted by disabling the transport feature of tonic_build as documented here and I've tested that this works as expected by using tonic_build directly.

@neoeinstein do you have any idea how I can disable this? Is it possible in the current implementation of protoc-gen-tonic?

I think #[path = "v1.account.rs"] better because autocomplete works

Plugin is failing with rust 1.65 stable, I am going to try and take a deeper look but any help would be great.

Failure: plugin prost: proto: cannot parse invalid wire-format data; context canceled; signal: killed; signal: killed; signal: killed; context canceled; context canceled; context canceled; signal: killed; signal: killed; signal: killed; context canceled
ERROR: buf generate proto stubs failed

Running any of the below commands all give the same error + exit code 1: "Minimal.rs: Tried to insert into file that doesn't exist."
protoc --tonic_out=. ./minimal.proto
protoc --prost-crate_out=. ./minimal.proto // actually gives "Cargo.toml: Tried to insert into file that doesn't exist."
protoc --prost-serde_out=. ./minimal.proto

The below commands work:
protoc --prost_out=. ./minimal.proto // produces Minimal.rs file with proto declaration but no GRPC service, as expected
protoc --tonic_out=. --tonic_opt=no_client=true,no_server=true ./minimal.proto // generates nothing
protoc --go-grpc_out=. ./min.proto // generates a go service as expected, just for example

Minimal proto that causes this issue:

syntax = "proto3";
package Minimal;
option go_package="test.minimal"

// The request message
message ExampleRequest {
  string name = 1;
}

// The response message
message ExampleReply {
  string message = 1;
}

// The service definition
service Example {
  // Declare one RPC
  rpc ExampleRpc (ExampleRequest) returns (ExampleReply) {}
}

OS: Ubuntu 20.04 LTS

protoc --version: libprotoc 3.6.1

cargo install --list | grep protoc:
protoc-gen-prost v0.1.3:
protoc-gen-prost
protoc-gen-prost-crate v0.1.5:
protoc-gen-prost-crate
protoc-gen-prost-serde v0.1.0:
protoc-gen-prost-serde
protoc-gen-tonic v0.1.0:
protoc-gen-tonic

So in the buf model you use buf to build everything, including the file descriptor sets (usually). But how you access the well known types was a bit of a mystery to me until I did some digging. In short you need to add the protoc-wkt crate and then use the file descriptor set like this:

use protoc_wkt::google::protobuf::{FILE_DESCRIPTOR_SET as GOOGLE_PROTOBUF_FILE_DESCRIPTOR_SET};

fn myfunc() {
    let reflection_service = tonic_reflection::server::Builder::configure()
        // Note: It is critical that the file descriptor set is registered for every
        // file that the API proto depends on recursively. If you don't, compilation
        // will still succeed but reflection will fail at runtime.
        //
        // TODO: Add a test for this / something in build.rs, this is a big footgun.
        .register_encoded_file_descriptor_set(API_V2_FILE_DESCRIPTOR_SET)
        .register_encoded_file_descriptor_set(TRANSACTION_V1_TESTING_FILE_DESCRIPTOR_SET)
        .register_encoded_file_descriptor_set(UTIL_TIMESTAMP_FILE_DESCRIPTOR_SET)
        .register_encoded_file_descriptor_set(GOOGLE_API_V1_FILE_DESCRIPTOR_SET)
        .register_encoded_file_descriptor_set(GOOGLE_PROTOBUF_FILE_DESCRIPTOR_SET)
        .build()
        .context("Failed to build reflection service")?;
}

It'd be great to have an example that demonstrates how to use them. If I get some time any time soon I can take a crack at doing it myself 🤠

I am having trouble using google.protobuf.Timestamp in my project. Is there documentation or an example showing how to configure my project to properly handle this protobuf type?

Hi! Learn a lot from this repo, really appreciate it.

Recently pbjson release 0.5.0 with preserve_proto_field_names option. I would like to know whether it is compatible with the whole system and if there has been any chance to bump the version up and support this feature.

If there is no further concern, I am willing to contribute to this feature.

I trying to generate code by protoc command in wsl2, but I getting:

If there is a proto file that has only service definitions like that ,

syntax = "proto3";

package b.a;

import "common.proto";

service BService {
  rpc Create(b.common.Request)
    returns (b.common.Response);

  rpc Get(b.common.Request)
    returns (b.common.Response);
}

the compilation will fail with this message .

b.a.rs: Tried to insert into file that doesn't exist.

A workaround is to add a type like that in the file.

enum TestEnum {
  ZERO = 0;
  ONE = 1;
  TWO = 2;
}

What happen I think is Prost doesn’t generate b.rs because it is empty so tonic cannot add its b.tonic.rs include to it.
Maybe it is related to issue #59.

Currently, protoc-gen-prost-crate leaves the insertion point comment as-is in the output Cargo.toml. It should probably remove it.

Here is the protobuf file:

// hello.proto
syntax = "proto3";
package hello;

service Greeter {
    rpc SayHello (HelloRequest) returns (HelloReply);
}

message HelloRequest {
    string name = 1;
}

message HelloReply {
    string message = 1;
}

I run the command:

protoc --prost_out=src/proto --tonic_out=src/proto  proto/hello.proto

Get file hello.rs and hello.tonic.rs. There is an error because it calls super::HelloRequest in hello.tonic.rs:

cannot find type `HelloRequest` in module `super` not found in `super`

I have to add use super::hello::*; to fix it manually although the hello.tonic.rs says:

// @generated
/// Generated client implementations.

/// Generated server implementations.

It seems I should not edit that file.
Am I using it in a wrong way? Or is it need to be improved?

@neoeinstein Hey, would it be possible to get a new release on crates.io that includes #17 ?

Hi, I'm trying to compile two unrelated proto files using protoc-gen-prost to create two separate modules but im finding that each separate invocation of protoc overwrites the generated lib.rs and Cargo.toml file rather than update in place.

My use case is a monorepo where the protos for many services are stored in /protos/ and the output goes in /generated/.

I have put together a minimal reproduction in this repo

You can see that I have two proto files:

• protos/foo/foo.proto
• protos/bar/bar.proto

I'm compiling these with the following commands in the script generate.sh:

protoc \
    --prost_out=generated/rust \
    --tonic_out=generated/rust \
    --prost-crate_out=generated/rust \
    --prost-crate_opt=gen_crate=templates/Cargo.toml \
    protos/foo/foo.proto

protoc \
    --prost_out=generated/rust \
    --tonic_out=generated/rust \
    --prost-crate_out=generated/rust \
    --prost-crate_opt=gen_crate=templates/Cargo.toml \
    protos/bar/bar.proto

and the generated lib.rs and Cargo.toml dont contain any references to Foo:

// @generated
pub mod proto {
    #[cfg(feature = "proto-bar")]
    // @@protoc_insertion_point(attribute:proto.bar)
    pub mod bar {
        include!("proto.bar.rs");
        // @@protoc_insertion_point(proto.bar)
    }
}

and

[package]
name = "protos"
version = "0.1.0"
edition = "2021"

[dependencies]
prost = "0.11.2"

[features]
default = []
# @@protoc_deletion_point(features)
# This section is automatically generated by protoc-gen-prost-crate.
# Changes in this area may be lost on regeneration.
proto_full = ["proto-bar"]
"proto-bar" = []
## @@protoc_insertion_point(features)

If I only run the first protoc command I can see that the foo insertions are generated correctly.

I think this is a bug and not a problem with a way I'm using the tool.

Thanks for your hard work here

I'm experimenting with this plugin with Bazel and found an issue where no files were being produced.

Below is a std::process::Command I'm using to invoke protoc

Command {
    program: "bazel-out/darwin_arm64-opt-exec-2B5CBBC6/bin/external/com_google_protobuf/protoc",
    args: [
        "bazel-out/darwin_arm64-opt-exec-2B5CBBC6/bin/external/com_google_protobuf/protoc",
        "--prost_out=bazel-out/darwin_arm64-fastbuild/bin",
        "--prost_opt=extern_path=.google.protobuf.Any=any_proto::google::protobuf::Any",
        "--prost_opt=extern_path=.google.protobuf.SourceContext=source_context_proto::google::protobuf::SourceContext",
        "--prost_opt=compile_well_known_types",
        "--plugin=protoc-gen-prost=bazel-out/darwin_arm64-opt-exec-2B5CBBC6/bin/external/rules_rust_prost__protoc-gen-prost-0.2.2/protoc-gen-prost__bin",
        "--proto_path=bazel-out/darwin_arm64-fastbuild/bin/external/com_google_protobuf/_virtual_imports/any_proto",
        "--proto_path=bazel-out/darwin_arm64-fastbuild/bin/external/com_google_protobuf/_virtual_imports/source_context_proto",
        "--proto_path=bazel-out/darwin_arm64-fastbuild/bin/external/com_google_protobuf/_virtual_imports/type_proto",
        "-Igoogle/protobuf/type.proto=bazel-out/darwin_arm64-fastbuild/bin/external/com_google_protobuf/_virtual_imports/type_proto/google/protobuf/type.proto",
        "-Igoogle/protobuf/any.proto=bazel-out/darwin_arm64-fastbuild/bin/external/com_google_protobuf/_virtual_imports/any_proto/google/protobuf/any.proto",
        "-Igoogle/protobuf/source_context.proto=bazel-out/darwin_arm64-fastbuild/bin/external/com_google_protobuf/_virtual_imports/source_context_proto/google/protobuf/source_context.proto",
        "bazel-out/darwin_arm64-fastbuild/bin/external/com_google_protobuf/_virtual_imports/type_proto/google/protobuf/type.proto",
    ],
}

I added some debug lines to https://github.com/neoeinstein/protoc-gen-prost/blob/protoc-gen-prost-v0.2.2/protoc-gen-prost/src/lib.rs#L87-L99 and found that whenever the first proto with a particular package is processed, all protos with the same package are ignored. In the case with the command above, I get the following output with the associated diff

diff --git a/protoc-gen-prost/src/lib.rs b/protoc-gen-prost/src/lib.rs
index 43fc5ba..3506ca9 100644
--- a/protoc-gen-prost/src/lib.rs
+++ b/protoc-gen-prost/src/lib.rs
@@ -86,14 +86,19 @@ impl ModuleRequestSet {
             |mut acc, (proto, raw)| {
                 let module = Module::from_protobuf_package_name(proto.package());
                 let proto_filename = proto.name();
+                eprintln!("________proto_filename = {}", proto_filename);
+                eprintln!("________module = {:?}", module);
                 let entry = acc.entry(module).or_insert_with(|| {
                     let mut request = ModuleRequest::new(proto.package().to_owned());
                     if input_protos.contains(proto_filename) {
+                        eprintln!("______checked {} in {:?}", proto_filename, input_protos);
                         let filename = match proto.package() {
                             "" => default_package_filename.to_owned(),
                             package => format!("{package}.rs"),
                         };
                         request.with_output_filename(filename);
+                    } else {
+                        eprintln!("_____checked {} not in {:?}", proto_filename, input_protos);
                     }
                     request
                 });

________proto_filename = google/protobuf/any.proto
________module = Module { components: ["google", "protobuf"] }
_____checked google/protobuf/any.proto not in {"google/protobuf/type.proto"}
________proto_filename = google/protobuf/source_context.proto
________module = Module { components: ["google", "protobuf"] }
________proto_filename = google/protobuf/type.proto
________module = Module { components: ["google", "protobuf"] }

This shows that any.proto, the first proto file defining package google.protobuf and having a ModuleRequest created for it while the rest are skipped, despite the input/target proto file being type.proto.

I'd greatly appreciate any guidance on how to fix this.

The currently released version blocks updating to the latest version of prost because it doesn't have 038cd34

Hey 👋

tonic_reflection seems broken when using the file descriptor sets generated when the api contains any google protobufs.

Got this error when trying to use grpcui.

Failed to compute set of methods to expose: Symbol not found: reporting.api.v1.ReportingService
caused by: File not found: google/protobuf/empty.proto

    Server::builder()
        .add_service(
            tonic_reflection::server::Builder::configure()
                .register_encoded_file_descriptor_set(grpc::reporting::api::v1::FILE_DESCRIPTOR_SET)
                .build()?,
        )

  - remote: buf.build/prost/plugins/prost:v0.2.1-1
    out: autogen/lang/rust
    opt:
      - bytes=.
      - compile_well_known_types
      - extern_path=.google.protobuf=::pbjson_types
      - file_descriptor_set
  - remote: buf.build/prost/plugins/serde:v0.2.1-1
    out: autogen/lang/rust
  - remote: buf.build/prost/plugins/tonic:v0.2.1-1
    out: autogen/lang/rust
    opt:
      - compile_well_known_types
      - extern_path=.google.protobuf=::pbjson_types
  - remote: buf.build/prost/plugins/crate:v0.3.1-1
    out: autogen/lang/rust
    opt:
      - no_features

tonic-reflection = "0.6.0"

I'm getting the following error using this plugin from rules_proto_grpc:

thread 'main' panicked at 'no source code info in request', external/crate-index__prost-build-0.11.1/src/code_generator.rs:56:14
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
--rust_prost_plugin_out: protoc-gen-rust_prost_plugin: Plugin failed with status code 101.

Bazel version is 5.3.0. Any reason this might be occuring?

max request message size is 4MB by default in tonic's 0.9.0 release hyperium/tonic#1337
currently, this size is not configurable from protoc-gen-tonic since it depends on tonic-build 0.8.x
a version bump should catch the codegen up

The protoc-gen-prost-serde documentation states the following: This tool supports all the same options from pbjson-build.

This is not correct. One example is ignore_unknown_fields, there could be more as well. The options in pbjson-build can be found here.

The code that sets these options can be found here:

Here's a short snippet of our proto file:

syntax = "proto3";

package issuer_api.v1;

import "common/v1/common.proto";

...

service IssuerService {
  rpc Authenticate(AuthenticateRequest) returns (AuthenticateResponse);
  rpc GetVerifier(GetVerifierRequest) returns (GetVerifierResponse);
  rpc Authorize(stream AuthorizeRequest) returns (stream AuthorizeResponse);
  rpc Issue(IssueRequest) returns (stream IssueResponse);
}

I generated the code via:

buf generate --path issuer_api

And the main file issuer_api.v1.rs only includes the tonic service definition included:

include!("issuer_api.v1.tonic.rs");

And also the generated code uses the dependency like this:

    pub authentication_signature: ::core::option::Option<super::super::common::v1::AuthenticationSignature>,

Instead of

    pub authentication_signature: ::core::option::Option<AuthenticationSignature>,

By including it

  - name: prost
    out: src/gen
    opt:
      - bytes=.
      - compile_well_known_types
      - extern_path=.google.protobuf=::pbjson_types
      - file_descriptor_set

This works great and I have reflection setup but when I call it it fails.

'google/protobuf/empty.proto' not found

tonic_reflection::server::Builder::configure()
        .register_encoded_file_descriptor_set(proto::name::FILE_DESCRIPTOR_SET)

It seems like pbjson-types should handle this and I see that it generates a bin file but I don't see a way to consume it. Am I missing something or is this a bug? Thanks

build/pbjson-types-7ae7be497a9998f0/out/proto_descriptor.bin

In general, these crates continue to generate code fine without relying on the newest version of Prost. However, the latest version of Prost adds some interesting new functionality into the generation, so we should look to update to work with that.

May also include updates to the latest version of tonic as well.

For build scripts and testing its important to understand what version of the plugin is being used.

protoc-gen-prost --version

ref.

$ protoc-gen-go --version
protoc-gen-go v1.28.0

My directory structure:

- root
  - api-crate
    - src
    | Cargo.toml
  - proto
    | buf.yaml
  | buf.gen.yaml
  | buf.work.yaml 

protoc-gen-prost-crate version v0.3.1

Now I run buf generate in the root directory, and it exits with error:

plugin prost-crate: The system couldn't find the given file. (os error 2)

buf.gen.yaml:

version: v1
plugins:
  - remote: buf.build/prost/plugins/prost:v0.2.1-1
    out: api-crate/src/gen
    opt:
      - compile_well_known_types
      - extern_path=.google.protobuf=::pbjson_types
      - file_descriptor_set
  - remote: buf.build/prost/plugins/serde:v0.2.1-1
    out: api-crate/src/gen
  - remote: buf.build/prost/plugins/tonic:v0.2.1-1
    out: api-crate/src/gen
    opt:
      - compile_well_known_types
      - extern_path=.google.protobuf=::pbjson_types
  - name: prost-crate
    out: api-crate
    strategy: all
    opt:
      - include_file=api-crate/src/gen/mod.rs
      - gen_crate

buf.gen.yaml

version: v1
plugins:
  - plugin: prost
    out: gen/src
    opt:
      - bytes=.
      - file_descriptor_set
  - plugin: prost-serde
    out: gen/src
  - plugin: prost-crate
    out: gen
    strategy: all
    opt:
      - gen_crate=Cargo.toml.tpl

buf generate

Failure: plugin prost-crate: could not find protoc plugin for name prost-crate - please make sure protoc-gen-prost-crate is installed and present on your $PATH

I have tried to run buf generate on the following repo https://github.com/edmondop/spark-connect-rust on the main branch and it fails with:

Failure: plugin buf.build/community/neoeinstein-tonic:v0.2.2: spark.connect.rs: does not exist

Unfortunately I haven't found a way to obtain more debug information:

❯ buf generate --debug
DEBUG   get_ref {"duration": "83.584µs"}
DEBUG   get_config      {"duration": "2.02425ms"}
DEBUG   get_workspace_config    {"duration": "324.583µs"}
DEBUG   get_config      {"duration": "18.417µs"}
DEBUG   get_config      {"duration": "2.292µs"}
DEBUG   get_module_config       {"duration": "1.5015ms"}
DEBUG   get_image       {"duration": "12.625µs"}
DEBUG   build   {"duration": "22.312125ms"}
DEBUG   build_module    {"duration": "22.313708ms"}
DEBUG   command {"duration": "657.238ms"}
Failure: plugin buf.build/community/neoeinstein-tonic:v0.2.2: spark.connect.rs: does not exist

while the -v option of buf generate doesn't produce any additional info. Could you please help?

pbjson-build 0.6 has an exclude option which is useful for providing customized serde impls, or just omitting serde for specified protobuf types. The serde plugin does not currently support this option.

docs.rs:

GitHub:

This can be reproduced on a fresh Ubuntu machine.

Repro steps

1. sudo apt update
2. sudo apt install curl build-essential -y
3. curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
4. source "$HOME/.cargo/env"
5. cargo install protoc-gen-prost

Current behavior:

error[E0603]: module `compiler` is private
  --> /home/larry/.cargo/registry/src/github.com-1ecc6299db9ec823/protoc-gen-prost-0.2.1/src/lib.rs:13:5
   |
13 |     compiler::{code_generator_response::File, CodeGeneratorRequest},
   |     ^^^^^^^^ private module
   |
note: the module `compiler` is defined here
  --> /home/larry/.cargo/registry/src/github.com-1ecc6299db9ec823/prost-types-0.11.7/src/lib.rs:15:1
   |
15 | mod compiler;
   | ^^^^^^^^^^^^

error[E0603]: module `compiler` is private
  --> /home/larry/.cargo/registry/src/github.com-1ecc6299db9ec823/protoc-gen-prost-0.2.1/src/generator.rs:3:18
   |
3  | use prost_types::compiler::{
   |                  ^^^^^^^^ private module
   |
note: the module `compiler` is defined here
  --> /home/larry/.cargo/registry/src/github.com-1ecc6299db9ec823/prost-types-0.11.7/src/lib.rs:15:1
   |
15 | mod compiler;
   | ^^^^^^^^^^^^

error[E0603]: module `compiler` is private
  --> /home/larry/.cargo/registry/src/github.com-1ecc6299db9ec823/protoc-gen-prost-0.2.1/src/generator/core.rs:2:18
   |
2  | use prost_types::compiler::code_generator_response::File;
   |                  ^^^^^^^^ private module
   |
note: the module `compiler` is defined here
  --> /home/larry/.cargo/registry/src/github.com-1ecc6299db9ec823/prost-types-0.11.7/src/lib.rs:15:1
   |
15 | mod compiler;
   | ^^^^^^^^^^^^

error[E0603]: module `compiler` is private
  --> /home/larry/.cargo/registry/src/github.com-1ecc6299db9ec823/protoc-gen-prost-0.2.1/src/generator/file_descriptor_set.rs:3:18
   |
3  | use prost_types::compiler::code_generator_response::File;
   |                  ^^^^^^^^ private module
   |
note: the module `compiler` is defined here
  --> /home/larry/.cargo/registry/src/github.com-1ecc6299db9ec823/prost-types-0.11.7/src/lib.rs:15:1
   |
15 | mod compiler;
   | ^^^^^^^^^^^^

error[E0282]: type annotations needed
  --> /home/larry/.cargo/registry/src/github.com-1ecc6299db9ec823/protoc-gen-prost-0.2.1/src/generator/core.rs:49:37
   |
49 |         request.write_to_file(move |buffer| {
   |                                     ^^^^^^
50 |             buffer.push_str("// @generated\n");
   |             ------ type must be known at this point
   |
help: consider giving this closure parameter an explicit type
   |
49 |         request.write_to_file(move |buffer: _| {
   |                                           +++

Some errors have detailed explanations: E0282, E0603.
For more information about an error, try `rustc --explain E0282`.
error: could not compile `protoc-gen-prost` due to 5 previous errors
warning: build failed, waiting for other jobs to finish...
error: failed to compile `protoc-gen-prost v0.2.1`, intermediate artifacts can be found at `/tmp/cargo-installZD0bjX

Expected behavior:

Install successfully.

Thanks a lot for writing this plugin, lovely stuff!

Is there a way to pass options to protoc-gen-prost if we don't use buf.

For example, based on protocolbuffers/protobuf#4759, using prost_opt should work:

COMMA_SEPARATED_PROST_CONFIG=compile_well_known_types,extern_path=.foo.bar.baz=::foo_bar_baz
protoc -I="$PROTO_DIR" \
    --prost_out="$OUTPUT_PATH" \
    --prost_opt="$COMMA_SEPARATED_PROST_CONFIG" \
    $PROTOC_PROTO_FILES_ARG

However, it seems like these options (prost_opt) are being ignored.

Thanks

I encountered an issue when trying to run the example code from the README. However, protoc-gen-prost appears to be working. I found there's a PR about similar issue but it's based on serde/tonic fix.

Steps to reproduce

env: Arch linux 2023‑03‑01, Rust 1.70.0, Cargo 0.2.2

1. clone the repo
2. cd into folder root
3. Install protoc-gen-prost with cargo install --path protoc-gen-prost
4. Install protoc-gen-prost-crate with cargo install --path protoc-gen-prost-crate
5. mkdir proto/gen
6. run protoc --prost-crate_out=proto/gen -I proto proto/greeter/v1/greeter.proto