It has been observed that the latest plugin will inadvertently output code for dependencies even if those dependencies were not specifically slated for compilation. This needs to be cleaned up.

Steps to reproduce

Compile any .proto file

Expected result

Successful production of Clojure output for the .proto

Observed result

The plugin crashes (see below)

protoc --clojure_out=grpc-client,grpc-server:src --proto_path=resources resources/addressbook.proto
Exception in thread "main" java.lang.NullPointerException
	at clojure.lang.RT.intCast(RT.java:1220)
	at protojure.plugin.main$_main.invokeStatic(main.clj:86)
	at protojure.plugin.main$_main.doInvoke(main.clj:80)
	at clojure.lang.RestFn.invoke(RestFn.java:397)
	at clojure.lang.AFn.applyToHelper(AFn.java:152)
	at clojure.lang.RestFn.applyTo(RestFn.java:132)
	at protojure.plugin.main.main(Unknown Source)
--clojure_out: protoc-gen-clojure: Plugin failed with status code 1.
make: *** [all] Error 1

The current handling of eliding default values when writing out to the wire interacts incorrectly with proto2 semantics.

As it stands right now, the plugin generates output files based on the number of input .protos. This doesn't work when multiple .proto files declare the same 'package'. The end result is that the plugin proposes multiple outputs with the same filename (since the filename is based on the package) and protoc rejects the output.

This common packaging is perfectly normal, and is utilized in two high-profile projects: Protobufs itself (via the Google "Well Known Types" and the Hyperledger Fabric project.

Therefore, our plugin needs to support this use case.

While attempting to protoc compile a bunch of .proto files, several times I encountered this message:

foo.proto is a proto3 file that contains optional fields, 
but code generator protoc-gen-clojure hasn't been updated to support optional fields in proto3. 
Please ask the owner of this code generator to support proto3 optional.

What is the prognosis/timeline for this, if any?
If not, any idea how much work it would be for someone who knows almost nothing about protobufs to add this?

Proto2 doesn't require enums to be 0-indexed, and therefore protojure generates incorrect behaviour for e.g. 1-indexed enums.

https://github.com/protocolbuffers/protobuf/blob/v3.12.0-rc1/docs/implementing_proto3_presence.md

If not, have you thought about how to do it?
Something like com.google.protobuf/Value->edn

In 0.9.3 when a Protocol Buffer message contains a field named: "values", all of the fields (except the "values" field) are missing in the generated code.

The message:

message A {
  string field1 = 1;
  string values = 2;
  string field3 = 3;
}

Produces:

(defrecord A-record [values]
  pb/Writer
  (serialize [this os]
    (serdes.core/write-String 2  {:optimize true} (:values this) os))
  pb/TypeReflection
  (gettype [this]
    "com.example.ab.A"))

(s/def :com.example.ab.A/values string?)
(s/def ::A-spec (s/keys :opt-un [:com.example.ab.A/values ]))
(def A-defaults {:values "" })

A similar message containing a field named: "values2"

message B {
  string field1 = 1;
  string values2 = 2;
  string field3 = 3;
}

Produces the correct record

(defrecord B-record [field1 values2 field3]
  pb/Writer
  (serialize [this os]
    (serdes.core/write-String 1  {:optimize true} (:field1 this) os)
    (serdes.core/write-String 2  {:optimize true} (:values2 this) os)
    (serdes.core/write-String 3  {:optimize true} (:field3 this) os))
  pb/TypeReflection
  (gettype [this]
    "com.example.ab.B"))

(s/def :com.example.ab.B/field1 string?)
(s/def :com.example.ab.B/values2 string?)
(s/def :com.example.ab.B/field3 string?)
(s/def ::B-spec (s/keys :opt-un [:com.example.ab.B/field1 :com.example.ab.B/values2 :com.example.ab.B/field3 ]))
(def B-defaults {:field1 "" :values2 "" :field3 "" })

Hello, is there anyway to parse and edn like {#xtdb/id \"1073c72f6e04ee66347a627efd9096910b790e66\" {:map {:k2 \"v2\"}, :bool? true, :number 3.0, :listing [1.0 2.0 3.0 4.0], :key \"ARROZ\", :crux.db/id :id1}}to google.protobuf.struct ?

Generated code:

;-----------------------------------------------------------------------------
; SpeculativeTxResponse
;-----------------------------------------------------------------------------
(defrecord SpeculativeTxResponse-record [valid-time tx-time tx-id entity-cache-size batch-size edn-document]
  pb/Writer
  (serialize [this os]
    (serdes.core/write-String 1  {:optimize true} (:valid-time this) os)
    (serdes.core/write-String 2  {:optimize true} (:tx-time this) os)
    (serdes.core/write-Int64 3  {:optimize true} (:tx-id this) os)
    (serdes.core/write-Int32 4  {:optimize true} (:entity-cache-size this) os)
    (serdes.core/write-Int32 5  {:optimize true} (:batch-size this) os)
    (serdes.core/write-embedded 6 (:edn-document this) os))
  pb/TypeReflection
  (gettype [this]
    "com.xtdb.protos.SpeculativeTxResponse"))

(s/def :com.xtdb.protos.SpeculativeTxResponse/valid-time string?)
(s/def :com.xtdb.protos.SpeculativeTxResponse/tx-time string?)
(s/def :com.xtdb.protos.SpeculativeTxResponse/tx-id int?)
(s/def :com.xtdb.protos.SpeculativeTxResponse/entity-cache-size int?)
(s/def :com.xtdb.protos.SpeculativeTxResponse/batch-size int?)

(s/def ::SpeculativeTxResponse-spec (s/keys :opt-un [:com.xtdb.protos.SpeculativeTxResponse/valid-time :com.xtdb.protos.SpeculativeTxResponse/tx-time :com.xtdb.protos.SpeculativeTxResponse/tx-id :com.xtdb.protos.SpeculativeTxResponse/entity-cache-size :com.xtdb.protos.SpeculativeTxResponse/batch-size]))
(def SpeculativeTxResponse-defaults {:valid-time "" :tx-time "" :tx-id 0 :entity-cache-size 0 :batch-size 0})

(defn cis->SpeculativeTxResponse
  "CodedInputStream to SpeculativeTxResponse"
  [is]
  (map->SpeculativeTxResponse-record (tag-map SpeculativeTxResponse-defaults (fn [tag index] (case index 1 [:valid-time (serdes.core/cis->String is)] 2 [:tx-time (serdes.core/cis->String is)] 3 [:tx-id (serdes.core/cis->Int64 is)] 4 [:entity-cache-size (serdes.core/cis->Int32 is)] 5 [:batch-size (serdes.core/cis->Int32 is)] 6 [:edn-document (com.google.protobuf/ecis->Struct is)] [index (serdes.core/cis->undefined tag is)])) is)))

(defn ecis->SpeculativeTxResponse
  "Embedded CodedInputStream to SpeculativeTxResponse"
  [is]
  (serdes.core/cis->embedded cis->SpeculativeTxResponse is))

(defn new-SpeculativeTxResponse
  "Creates a new instance from a map, similar to map->SpeculativeTxResponse except that
  it properly accounts for nested messages, when applicable.
  "
  [init]
  {:pre [(if (s/valid? ::SpeculativeTxResponse-spec init) true (throw (ex-info "Invalid input" (s/explain-data ::SpeculativeTxResponse-spec init))))]}
  (-> (merge SpeculativeTxResponse-defaults init)
      (cond-> (some? (get init :edn-document)) (update :edn-document com.google.protobuf/new-Struct))
      (map->SpeculativeTxResponse-record)))

(defn pb->SpeculativeTxResponse
  "Protobuf to SpeculativeTxResponse"
  [input]
  (cis->SpeculativeTxResponse (serdes.stream/new-cis input)))

(def ^:protojure.protobuf.any/record SpeculativeTxResponse-meta {:type "com.xtdb.protos.SpeculativeTxResponse" :decoder pb->SpeculativeTxResponse})

Google code: com.google.protobuf/ecis->Struct

;-----------------------------------------------------------------------------
; Struct
;-----------------------------------------------------------------------------
(defrecord Struct-type [fields]
  pb/Writer

  (serialize [this os]
    (serdes.complex/write-map new-Struct-FieldsEntry 1 (:fields this) os)))

(s/def ::Struct-spec (s/keys :opt-un []))
(def Struct-defaults {:fields [] })

(defn cis->Struct
  "CodedInputStream to Struct"
  [is]
  (->> (tag-map Struct-defaults
         (fn [tag index]
             (case index
               1 [:fields (serdes.complex/cis->map ecis->Struct-FieldsEntry is)]

               [index (serdes.core/cis->undefined tag is)]))
         is)
        (map->Struct-type)))

(defn ecis->Struct
  "Embedded CodedInputStream to Struct"
  [is]
  (serdes.core/cis->embedded cis->Struct is))

(defn new-Struct
  "Creates a new instance from a map, similar to map->Struct except that
  it properly accounts for nested messages, when applicable.
  "
  [init]
  {:pre [(if (s/valid? ::Struct-spec init) true (throw (ex-info "Invalid input" (s/explain-data ::Struct-spec init))))]}
  (-> (merge Struct-defaults init)
      (map->Struct-type)))

(defn pb->Struct
  "Protobuf to Struct"
  [input]
  (cis->Struct (serdes.stream/new-cis input)))

Reference:

https://developers.google.com/protocol-buffers/docs/proto3#any

As it stands right now, the code generator emits both client and server code into one namespace. This results in conflicts that was resolved by adding a "call-" prefix to the client side code. This isn't ideal. Suggest splitting the client and server namespaces to avoid the conflict and thus retain the natural function names.

We're trying to use a protoc-gen-clojure-gen'd lib in our application, and the top-level namespace require includes the namespace itself. This doesn't cause problems on first load, but does cause problems when we try to reload the namespaces as a part of hot-reloading functionality.

Should the namespace be requiring itself? If so, why is it necessary (for my own edification)? If not, is that something that can safely be removed?

I can provide an example if desired.

https://github.com/protojure/protoc-plugin/releases

The latest protojure-lib has dropped support for size-XX operations in preparation for ClojureScript support. This patch ports the plugin to the latest protojure-lib by adjusting for the new namespaces and eliminating the size-XX calls

Description

I'm trying to use the buf project to build a protobuf client that's hosted with their system. When I use protoc-gen-clojure as the target plugin, the command fails with:

Failure: plugin clojure: fork/exec /Users/me/.local/bin/protoc-gen-clojure: exec format error

This doesn't seem to be specific to buf, but rather to the packaging of the plugin:

package main

import (
	"fmt"
	"os/exec"
)

func main() {
	fmt.Println(exec.Command("protoc-gen-clojure").Run())
}

produces the same error.

This happens with both the prebuilt 2.1.0 release and with a locally built binary based on most recent master.

Is there anything about jars that makes them incompatible with a fork call? If not, what would need to change to make this compatible with go tooling? It'd be nice to be able to use this plugin with buf.

I am often wondering what binary my various systems have installed. Proposal is to add a --version switch, similar to Chaintool

Can you please include a test configuration for the resulting template? I have been trying to create a test but doesn't matter what I send it always return an empty body

In 0.9.3 if two messages (A and B) have oneof fields with the same name (in this case: my_oneof), the generated code contains a single convert-my-oneof function for message: A but no convert-* function for message: B.

new-B calls A's convert-my-oneof function resulting in a B-record instance with missing fields.

syntax = "proto3";
package com.example.ab;

message A {
  oneof my_oneof {
    string s = 1;
    int32 i = 2;
  };
}

message B {
  oneof my_oneof {
    bool b = 1;
    float f = 2;
  };
}

Produces:

(defn convert-my-oneof [origkeyval]
  (cond
     (get-in origkeyval [:my-oneof :s]) origkeyval
     (get-in origkeyval [:my-oneof :i]) origkeyval
     :default origkeyval))

(defn new-A
  "Creates a new instance from a map, similar to map->A except that
  it properly accounts for nested messages, when applicable.
  "
  [init]
  {:pre [(if (s/valid? ::A-spec init) true (throw (ex-info "Invalid input" (s/explain-data ::A-spec init))))]}
  (-> (merge A-defaults init)
      (convert-my-oneof)
      (map->A-record)))

(defn new-B
  "Creates a new instance from a map, similar to map->B except that
  it properly accounts for nested messages, when applicable.
  "
  [init]
  {:pre [(if (s/valid? ::B-spec init) true (throw (ex-info "Invalid input" (s/explain-data ::B-spec init))))]}
  (-> (merge B-defaults init)
      (convert-my-oneof)
      (map->B-record)))

Depends on protojure/lib#4

diff --git a/resources/testdata/kitchensink.proto b/resources/testdata/kitchensink.proto
index 4b6e013..398a6c4 100755
--- a/resources/testdata/kitchensink.proto
+++ b/resources/testdata/kitchensink.proto
@@ -78,7 +78,7 @@ message OneOf {
         string s = 2;
         SimpleString ss = 3;
     }
-    oneof AndAnotherOne {
+    oneof AndAnother_One {
         string aas = 4;
         bool aab = 5;
         OEnum aae = 6;

leading to:

(defn write-AndAnother-One [AndAnother-One os]
  (let [field (first AndAnother-One)
        k (when-not (nil? field) (key field))
        v (when-not (nil? field) (val field))]
     (case k
         :aas (serdes.core/write-String 4  {:optimize false} v os)
         :aab (serdes.core/write-Bool 5  {:optimize false} v os)
         :aae (write-OEnum 6  {:optimize false} v os)
         nil)))
...
(defrecord OneOf-record [firsts One AndAnother-One seconds eo num e FinalOne]
  pb/Writer
  (serialize [this os]
    (serdes.core/write-String 1  {:optimize true} (:firsts this) os)
    (write-One  (:One this) os)
    (write-AndAnother_One  (:AndAnother-One this) os)
    (serdes.core/write-embedded 7 (:seconds this) os)
    (serdes.core/write-embedded 8 (:eo this) os)
    (serdes.core/write-Int32 9  {:optimize true} (:num this) os)
    (write-OEnum 10  {:optimize true} (:e this) os)
    (write-FinalOne  (:FinalOne this) os))
  pb/TypeReflection
  (gettype [this]
    "com.example.kitchensink.OneOf"))

this is probably an easy patch in the parse.oneof ns, but theres probably a way to hoist the clojurifying so that it only needs to be done once

Given a foo.proto importing another bar.proto, if bar.proto uses a go_package option, the require of the bar ns in the foo ns will be output as [ :as ]

This is either related to our not ignoring go_package, or the CodeGenRequest including the go_package under a separate key that we don't ignore or handle properly.

Hi, i'm having a problem with Clojure as a server and Golang as a client.
When my proto file contains the following lines:

syntax = "proto3";
package service.example.client;
option java_package = "br.com.service.example.client";

When we generate the server code, the generated code namespaces follows the java_package option, as expected.
However, the Golang client generated code doesn't follow the java_package naming convention, and instead, it tries to reach for methods declared on package service.example.client;

Because there's no namespaces declared as described above, the client fails to communicate with the server, and tells us that it couldn't find the implementation of the requested method.

To reproduce the problem, just follow the instructions on this repo: https://github.com/matheusfrancisco/issue-protoc-protojure-grpc

Would I like to know if his a bug on protojure side, if that's the case. I would like to help open a Pull Request as soon as possible?

Project README.md and https://protojure.github.io/

proto "import" statements refer to the proto "package", but the resulting import may have overridden the package with a language pragma, such as "java_package". This results in an incongruence between the generated :requires statement in the dependent code and the (ns) of the target.