clojusc / protobuf Goto Github PK

As far as I understand this library, it allows only to read protos from and into bytes. I would be useful to also add text proto support which is sometimes used for small config files.

It might look like this:

person {
  name: "John Doe"
  email: "[email protected]"
}

or this:

person <
  name: "John Doe"
  email: "[email protected]"
>

• Update protocol
• Add docs page (markdown) showing how to use it

Add profiles for subsequent Clojure versions. Also update the testall alias to include these.

Update: The example below has been edited to work with the most recent API.

Looks like we have a missing feature in the Clojure API ... (probably the Java wrapper?)

To see this problem, just fire up the REPL and:

[protobuf.dev] λ=> (require '[protobuf.core :as protobuf])
nil
[protobuf.dev] λ=> (import '(protobuf.examples.tutorial AddressBookProtos$AddressBook))
protobuf.examples.tutorial.AddressBookProtos$AddressBook
[protobuf.dev] λ=> (def AddressBook (protobuf/create AddressBookProtos$AddressBook))
#'protobuf.dev/AddressBook
[protobuf.dev] λ=> (pprint (protobuf/->schema AddressBook))
{:type :struct,
 :name "tutorial.AddressBook",
 :fields
 {:people
  {:type :list,
   :values
   {:type :struct,
    :name "tutorial.Person",
    :fields
    {:name {:type :string},
     :id {:type :int},
     :email {:type :string},
     :phones
     {:type :list,
      :values
      {:type :struct,
       :name "tutorial.Person.PhoneNumber",
       :fields
       {:number {:type :string},
        :type
        {:type :enum,
         :values #{:home :work :mobile},
         :default
         #object[com.google.protobuf.Descriptors$EnumValueDescriptor 0x1a07f7ad "home"]}}}}}}}}}
nil
[protobuf.dev] λ=> (get-in (protobuf/->schema AddressBook) 
                           [:fields :people :values 
                            :fields :phones :values 
                            :fields :type :default])
#object[com.google.protobuf.Descriptors$EnumValueDescriptor 0x1a07f7ad "home"]

This is coming from the following proto file definition:

syntax = "proto2";

package tutorial;

option java_package = "protobuf.examples.tutorial";
option java_outer_classname = "AddressBookProtos";

message Person {
  required string name = 1;
  required int32 id = 2;
  optional string email = 3;

  enum PhoneType {
    MOBILE = 0;
    HOME = 1;
    WORK = 2;
  }

  message PhoneNumber {
    required string number = 1;
    optional PhoneType type = 2 [default = HOME];
  }

  repeated PhoneNumber phones = 4;
}

message AddressBook {
  repeated Person people = 1;
}

Many of the included deps are deprecated, old, or simply used for a single function or two. Either work around the dep, or provide a simple implementation that provides the most minimal possible functionality.

• org.flatland/useful
• org.flatland/schematic
• ordered-collections (https://github.com/ninjudd/ordered-collections)
• org.flatland/io
• gloss

• Adjoin -> Combiner
• adjoin-onto -> combine-onto
• adjoin -> combine
• protobuf? -> map?
• protodef? -> mapdef?
• protodef -> mapdef
• protobuf/protobuf -> protobuf/create
• protobuf-load and protobuf-load-stream -> parse
• protobuf-dump ->->bytes
• protobuf-seq -> read
• protobuf-write -> write

Hey folks, I am evaluating this and testing some simple scenarios of mutation. What I am trying to test is:

1. creating a proto object
2. mutate in some Clojure map-related API (that is allowed by the schema)
3. serialize to bytes
4. deserialize and see that the mutation actually worked:

now, given this schema:

syntax = "proto2";

package clojure_protobuf_poc.core.person;

option java_outer_classname = "PersonProto";

message Name {
  optional string name = 1;
  optional string surname = 2;
}

message Person {
  optional int32  id    = 1;
  optional Name name  = 2;
  optional string email = 3;
  repeated string likes = 4;
}

and this test code:

(ns clojure-protobuf-poc.core-test
  (:require [clojure.test :refer [testing deftest is]]
            [protobuf.core :as protobuf])
  (:import [clojure_protobuf_poc.core.person PersonProto$Name PersonProto$Person]
           [protobuf.impl.flatland.core FlatlandProtoBuf]))

(defn bytes->Person [person-bytes]
  (protobuf/bytes-> (protobuf/create PersonProto$Person) person-bytes))

(defn mutate->round-trip [pb f]
  (let [mutated (f pb)]
    (is (= FlatlandProtoBuf (type mutated)))
    (-> mutated 
        protobuf/->bytes 
        bytes->Person)))

(deftest mutation
  (let [alice-name (protobuf/create PersonProto$Name {:name "Alice" :surname "Cohen"})
        alice (protobuf/create PersonProto$Person {:id 108 :name alice-name :email "[email protected]"})]
    (println alice)
    (testing "no mutation"
      (let [alice-tag (mutate->round-trip alice identity)]
        (is (= alice-tag alice))))
    (testing "dissoc"
      (let [no-id (mutate->round-trip alice #(dissoc % :id))]
        (is (= nil (:id no-id)))))
    (testing "assoc-in"
      (let [new-surname (mutate->round-trip alice #(assoc-in % [:name :surname] "Levi"))]
        (is (= "Levi" (get-in new-surname [:name :surname])))))
    (testing "update-in"
      (let [no-surname (mutate->round-trip alice #(update-in % [:name] dissoc :surname))]
        (is (= nil (get-in no-surname [:name :surname]))))) ; fails-> we get "Cohen"
    (testing "assoc"
      (let [new-email1 (mutate->round-trip alice #(assoc % :email "new-email"))]
        (is (= "new-email" (:email new-email1)))))
    (testing "merge" 
      ; crashes cause after the merge we get  java.lang.IllegalArgumentException: No implementation of 
      ; method: :->bytes of protocol: #'protobuf.core/ProtoBufAPI 
      ; found for class: protobuf.PersistentProtocolBufferMap
      (let [new-email2 (mutate->round-trip alice #(merge % {:email "new-email"}))]
        (is (= "new-email" (:email new-email2)))))))

tests fail with:

FAIL in (mutation) (core_test.clj:31)
update-in
expected: nil
  actual: "Cohen"
    diff: + "Cohen"

lein test :only clojure-protobuf-poc.core-test/mutation

FAIL in (mutation) (core_test.clj:12)
merge
expected: protobuf.impl.flatland.core.FlatlandProtoBuf
  actual: protobuf.PersistentProtocolBufferMap
    diff: - protobuf.impl.flatland.core.FlatlandProtoBuf
          + protobuf.PersistentProtocolBufferMap

lein test :only clojure-protobuf-poc.core-test/mutation

ERROR in (mutation) (core_deftype.clj:583)
Uncaught exception, not in assertion.
expected: nil
  actual: java.lang.IllegalArgumentException: No implementation of method: :->bytes of protocol: #'protobuf.core/ProtoBufAPI found for class: protobuf.PersistentProtocolBufferMap
 at clojure.core$_cache_protocol_fn.invokeStatic (core_deftype.clj:583)
...
...

Can you explain why update-in and merge aren't supported and what I (probably) have done wrong?
Thanks!

Maybe, put differently, it looks like PersistentProtocolBufferMap isn't 100% compatible with a Clojure map:

1. Is that correct?
2. If so, should I strive to convert it into a Clojure map? What is the fastest way to do so (recursively)?

The work done on #1 (removal of as many dependencies as possible) resulted in a pause with regard to these libraries:

• org.flatland/io
• gloss

It would be nice to not depend upon these old libraries that seem to be doing an enormous amount of work ... work that seems to be overly complex.

If perfect generality wasn't striven for, what might we end up with? How much more simple might the code be, with fewer deps, and maybe even increased efficiency?

Note that both glass and ordered-collections are abandon-ware at this point in time ...

Using the Photo example, I added a capture_by field which is oneof either a Phone or Camera.

syntax = "proto3";
package protobuf.examples.photo;

import "protobuf/extensions.proto";

option java_package = "examples";
option java_multiple_files = true;

message Photo {
  int32 id  = 1;
  string path = 2;
  repeated Label labels = 3 [(set) = true];
  repeated Attr attrs = 4 [(map) = true];
  repeated Tag tags  = 5;
  bytes image = 6;
  Type type = 7;
  oneof captured_by {
    Camera camera = 8;
    Phone phone = 9;
  }

  enum Type {
    JPEG = 0;
    PNG = 1;
    GIF = 2;
  }

  message Label {
    string item = 1;
    bool exists = 2;
  }

  message Attr {
    string key = 1;
    string val = 2;
  }

  message Tag {
    int32 person_id = 1;
    double x_coord = 2;
    double y_coord = 3;
    int32 width = 4;
    int32 height = 5;
  }

  message Camera {
    string model = 1;
    string make = 2;;
  }

  message Phone {
    string model = 1;
    string make = 2;
  }
}

The schema does not seem to indicate this relationship:

user> (proto/schema Photo)
{:type :struct,
 :name "protobuf.examples.photo.Photo",
 :fields
 {:path {:type :string},
  :tags
  {:type :list,
   :values
   {:type :struct,
    :name "protobuf.examples.photo.Photo.Tag",
    :fields
    {:person-id {:type :int},
     :x-coord {:type :double},
     :y-coord {:type :double},
     :width {:type :int},
     :height {:type :int}}}},
  :labels {:type :set, :values {:type :string}},
  :phone
  {:type :struct,
   :name "protobuf.examples.photo.Photo.Phone",
   :fields {:model {:type :string}, :make {:type :string}}},
  :type {:type :enum, :values #{:png :gif :jpeg}},
  :id {:type :int},
  :camera
  {:type :struct,
   :name "protobuf.examples.photo.Photo.Camera",
   :fields {:model {:type :string}, :make {:type :string}}},
  :attrs
  {:type :map, :keys {:type :string}, :values {:type :string}},
  :image {:type :byte_string}}}

But creating the protobuf does seem to respect this relationship (somewhat):

user> (proto/create Photo {:phone {:model "model" :make "make"} :camera {:model "model" :make "make"}})
{:camera {:model "model", :make "make"}}

Oddly enough it does not seem to retain the captured_by field and instead brings both fields to the top-level of the schema. It does only allow setting one of these fields

The following needs to be made to work:

(with-open [writer (io/writer "/tmp/address-book.db")]
  (protobuf/write writer addresses))

How can extensions be accessed?
Say, a provided proto is defined with: 'extend google.protobuf.FieldOptions'
(not the predefined map/map-by/meta etc)

How can one pull that info?

Whoever you are, I have been looking for an updated version of this library for ages. if you could put an ETH/devtip/Open Collective I would love to send you a tip.

Tasks:

• Add proto3 examples
• Add proto3 tests
• Add proto3 support for default value in enums - #30
• Add support for extracting the syntax of any given message
• Update Travis CI config to support proto3 on Travis builds (Travis Ubuntu containers currently only support 3)

Hi, is there a way to somehow convert from and to a native protobuf object?

I want to (->
read native protobuf objects from parquet ( via org.apache.parquet.proto/ProtoReadSupport)
Transform them into clojusc/protobuf based objects
Manipulate them as edn via this promising lib :)
Build native protobuf objects from the transformed values
Write to parquet ( via ProtoWriteSupport )
)

I am not sure if this approach is sound, but currently I see no way to convert to/from the native protobuf object.
In other words, the following blog/project demonstrates the approach, and it even references this library(in its older form?) as an alternative. But it will not really work without some conversion - as it rely on Kryo serialisation and really expects the native protobuf object and not the wrapper object:
https://adambard.com/blog/parquet-protobufs-spark/
https://github.com/adambard/sparkquet/blob/master/src/clj/sparkquet/core.clj

The actual objects are highly nested/repeated and manipulating them as edn would be much simpler.
Sorry if I am missing something basic here.

See: https://developers.google.com/protocol-buffers/docs/javatutorial

I want to put some message to Any.
but the problem occurs.

let me assume. we have that kind of message.

message A {
   Any any;
}

messsage B {
   int b;
}

So I can writing small encode/decode part for B

(let [message-b {:b 10}
      encoded (->> (protobuf/create B message-b)
                   (protobuf/->bytes)
                   (assoc {} :value)
                   (protobuf/create Any))
      decoded (->> encoded
                   :value
                   (protobuf/bytes-> (protobuf/create B))
                   (into {}))]
  (= message-b decoded))

but When I combine A with B . there is some tricky problem.

(let [message-b {:b 10}
      encoded (->> (protobuf/create B message-b)
                   (protobuf/->bytes)
                   (assoc {} :value)
                   (protobuf/create Any)
                   (assoc {} :any)
                   (protobuf/create A)
                   (protobuf/->bytes))

      decoded (->> encoded
                   (protobuf/bytes-> (protobuf/create A))
                   :any
                   (protobuf/create A)
                   :value
                   ;; at this point. value's type is bytestring so It need to call `.toByteArray`
                   (.toByteArray)       ; <<< a little tricky
                   (protobuf/bytes-> (protobuf/create B))
                   (into {}))]
  (= message-b decoded))

So I need to fix encode/decode part like that.

(let [message-b {:b 10}
      encoded (->> (protobuf/create B message-b)
                   (protobuf/->bytes)
                   ;; I inserted toByteArray on decoder. but it's type is `byte[]`
                   (com.google.protobuf.ByteString/copyFrom) ;; So It need to convert to bytestring.
                   (assoc {} :value)
                   (protobuf/create Any))
      decoded (->> encoded
                   :value
                   (.toByteArray) ;; so I inserted.
                   (protobuf/bytes-> (protobuf/create B))
                   (into {}))]
  (= message-b decoded))

It's works.
but com.google.protobuf.ByteString/copyFrom on this code likes tricky skill.
Is any other good suggestion of this? or Is anything what i missed?

Just use protobuf.* instead.

Sometime in the last week, Travis CI changed the default container setup we've been using for CI/CD on this project. Last week, everything built find; this morning, I'm seeing these errors on Travis:

This job is running on container-based infrastructure, which does not allow use of 'sudo', setuid, and setgid executables.
If you require sudo, add 'sudo: required' to your .travis.yml

I've done this before, I just have to look at the Travis docs to get what we need.

As a codox markdown topic, from the project's original README.

Thanks for the project, really useful and overall nice to work with.
I have noticed a bug regarding default values. For an enum, if it's set to a default value, then it will not show up in the clojure structure, and when deserializing it, the resulting object will not have any value for the enum.

I have created a simple test case exhibiting the problem at https://github.com/geekingfrog/clojure-protobuf-bug-default-enum
There is a java reference which shows the correct behavior, and a simple clojure file to compare.

The ghaskins/clojure-protobuf fork removed profiles that are required for testing. Re-add these in order to use the testall alias and test on previous releases of Clojure.

Missing profiles:

  :profiles {:1.3 {:dependencies [[org.clojure/clojure "1.3.0"]]}
             :1.5 {:dependencies [[org.clojure/clojure "1.5.0-master-SNAPSHOT"]]}
             :dev {:dependencies [[gloss "0.2.1"]]}}

Following on from the exploration done in #18, we're going to create a new API for end users of this library.

Tasks:

• Move current code from protobuf.core to protobuf.impl.flatland.*
• Improve the organization of the code that's moved to protobuf.impl.flatland.*
• Create a protocol in protobuf.core and pull in behaviours and records for extend from impl.*
• In protobuf.impl.flatland.core just have a FlatlandProtoBuf record and its corresponding behaviour def
• Have ProtoBuf implementation records take a compiled Java protobuf class (inner) as a field, e.g.,:

    (->MyProtoBuf :protobuf-class Example$Person)

• Figure out a static method approach for schema rendering
• Display instance data when printing protobuf impl in the REPL
• Improve usability of protobuf impl with core seq and map functions
• Update the docs to reflect all these changes

See protobuf.core, for example.

When you define a protobuf with enums such as "HOME", "WORK", and "MOBILE", protobuf represents these as :home, :work, and :mobile. However, a null pointer exception occurs if you try to use the keywords when creating a new message; if the protobuf was defined with a string in all caps, that's what needs to be used when creating a message, regardless of representation.

The ambiguity arising from this is not good -- it should be one or the other (i.e., don't convert, or allow keywords to be used when creating a message).

I guess this is going to end up as part of #16, but it kinda deserves a ticket of its own.

Can we provide the naming strategies in straight-up Clojure? Efficiently? Can we ditch this?

What are the set of operations the majority of users will perform? How do these map to the current implementation? Are the function names logical and consistent? Do they seem natural for novice and expert alike? Are they all in the same namespace? Would it make sense for them to be in the same namespace? Or to provide an API namespace that pulls them from logically separate areas of the codebase?

We noticed funny behaviour when trying to deserialize a byte array that was generated from a proto. I understand that default values don't get serialized, and hence deserializing them is not possible. However, I believe that setting up a default value when converting a bytearray to a map would be immensely helpful, instead of having no keys at all and that resulting in a nil.

Let me illustrate this with an example: Assuming you have a Person proto. Considering we are using proto3, I will not be using required and optional.

package your.namespace.person;

option java_outer_classname = "Example";

message Person {
  int32  id    = 1;
  string name  = 2;
  string email = 3;
  repeated string likes = 4;
}

Now given I have populated a protobuf map of type Person using
{ :id 123 :name "test subject"}

and serialized it to bytes; Once I try to deserialize it, I get a hashmap of
{:id 123 :name "test subject"}

I would instead expect default values to be populated on the resultant map. So; on deserializing, I would expect
{ :id = 123; :name = "test subject" :email = <empty string> :likes = <empty list> }

Is there something I am missing or is this desired behaviour? If it is something that is by design, would you be happy to accept a patch that allows us to populate default values for the fields not sent on the wire?

A common pattern for Clojure projects that also support a lein plugin is to combine projects into a single repo. Let's do that.

Make sure that the complete commit history from lein-protobuf is included.