Giter VIP home page Giter VIP logo

protocol-buffers-schema's Introduction

protocol-buffers-schema

No nonsense protocol buffers schema parser written in Javascript

npm install protocol-buffers-schema

build status

Usage

First save the following file as example.proto

syntax = "proto2";

message Point {
  required int32 x = 1;
  required int32 y=2;
  optional string label = 3;
}

message Line {
  required Point start = 1;
  required Point end = 2;
  optional string label = 3;
}

The run the following example

var fs = require('fs')
var schema = require('protocol-buffers-schema')

// pass a buffer or string to schema.parse
var sch = schema.parse(fs.readFileSync('example.proto'))

// will print out the schema as a javascript object
console.log(sch)

Running the above example will print something like

{
  syntax: 2,
  package: null,
  enums: [],
  messages: [{
    name: 'Point',
    enums: [],
    messages: [],
    options: {},
    fields: [{
      name: 'x',
      type: 'int32',
      tag: 1,
      required: true,
      repeated: false,
      options: {}
    }, {
      name: 'y',
      type: 'int32',
      tag: 2,
      required: true,
      repeated: false,
      options: {}
    }, {
      name: 'label',
      type: 'string',
      tag: 3,
      required: false,
      repeated: false,
      options: {}
    }]
  }, {
    name: 'Line',
    enums: [],
    messages: [],
    options: {},
    fields: [{
      name: 'start',
      type: 'Point',
      tag: 1,
      required: true,
      repeated: false,
      options: {}
    }, {
      name: 'end',
      type: 'Point',
      tag: 2,
      required: true,
      repeated: false,
      options: {}
    }, {
      name: 'label',
      type: 'string',
      tag: 3,
      required: false,
      repeated: false,
      options: {}
    }]
  }],
  options:{}
}

API

schema.parse(protobufSchemaBufferOrString)

Parses a .proto schema into a javascript object

schema.stringify(schema)

Stringifies a parsed schema back into .proto format

License

MIT

protocol-buffers-schema's People

Contributors

adon-at-work avatar aliksend avatar aoj avatar astro avatar bydga avatar cpdeethree avatar davidbruant avatar hvoecking avatar iwasinnam avatar kyle-schuetz avatar litichevskiydv avatar mafintosh avatar mkotelnikov avatar natevw avatar nxtn avatar pgirolami avatar raydog avatar rthewhite avatar sastan avatar wanglam avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

protocol-buffers-schema's Issues

Error: Unexpected token: "XXX.proto". Expected ";"

We do not support import public when running grpc-dynamic-gateway.

After removing the public, it runs without any problem.

/home/huan/git/grpc-dynamic-gateway/node_modules/protocol-buffers-schema/parse.js:501
  if (tokens[0] !== ';') throw new Error('Unexpected token: ' + tokens[0] + '. Expected ";"')
                         ^

Error: Unexpected token: "puppet/base.proto". Expected ";"

Related issue: grpc-ecosystem/grpc-gateway#139
Link to konsumer/grpc-dynamic-gateway#50

can't specify multiple schemes for openapiv2_swagger

protobuff like this

option (grpc.gateway.protoc_gen_swagger.options.openapiv2_swagger) = {
  info: {
    title: "Example code"
    version: "1.0"
    contact: {
      name: "project"
      url: "https://github.com/codesprut/project"
      email: "[email protected]"
    }
  }
  schemes: HTTP
  schemes: HTTPS
  consumes: "application/json"
  produces: "application/json"
}

Produce:

Error: Duplicate option map key schemes
    at onoptionMap (\node_modules\protocol-buffers-schema\parse.js:459:43)
    at onoption (\node_modules\protocol-buffers-schema\parse.js:421:19)
    at Function.parse (\node_modules\protocol-buffers-schema\parse.js:664:19)

Decimal default value not parsed correctly

message tester { 
  optional float number1 = 1 [default = 1.79769];
  optional string string1 = 2;
}

Encoding and decoding { string1: "" } yields:

{ number1: 1, string1: '' }

Unexpected token in message field

i am unable to parse a .proto file

it throws an error ('Unexpected token in message field: ' + tokens[0])

can anybody say what does this error mean and how to fix?

Expose optional keyword for proto3

Hi @mafintosh - optional in proto3 has the meaning that the absence of the field can be distinguished from its default value. At the moment this module is swallowing all occurrences of the optional keyword so that users cannot see when fields have been marked this way.

I would suggest adding an optional boolean beside required and repeated. required would still always be false for proto3.

Would you accept a PR with this change ?

Support syntax = "proto2"

Google's protobuf implementation since v3 recommends that all protobuf files use the line syntax = "proto2" or syntax = "proto3" to indicate which version of protocol buffers is being used. (Version 2 was the original version of protocol buffers that Google open-sourced.)

See: https://github.com/google/protobuf/releases/tag/v3.0.0-alpha-2

Unfortunately, protobuf-schema cannot handle this line:

Error: Unexpected token: syntax
    at Function.parse (/usr/local/lib/node_modules/protocol-buffers/node_modules/protobuf-schema/parse.js:351:15)
    at module.exports (/usr/local/lib/node_modules/protocol-buffers/index.js:9:85)
    at repl:1:24
    at REPLServer.defaultEval (repl.js:124:27)
    at bound (domain.js:254:14)
    at REPLServer.runBound [as eval] (domain.js:267:12)
    at REPLServer.<anonymous> (repl.js:277:12)
    at emitOne (events.js:77:13)
    at REPLServer.emit (events.js:166:7)
    at REPLServer.Interface._onLine (readline.js:195:10)

Obviously supporting proto3 is a larger endeavor, but protobuf-schema should definitely accept proto2 and proceed.

Reserved fields

When using protobuff it's an best practice that when you remove fields from a message that you reserve the tag number to prevent issues when adding a new field again. As described in the language guide: https://developers.google.com/protocol-buffers/docs/proto

message Foo {
  reserved 2, 15, 9 to 11;
  reserved "foo", "bar";
}

Currently the parser break on this, would like to make a pull request to fix this. But i'm in doubt what the proper solution would be. Should it just skip over the reserved tags and leave them out of the JSON completely or should it parse the tags properly and mark them in the JSON somewhere as reserved?

Scientific notation is not handled

For an example, given a protocol schema:

message testmsg { 
  optional double number1 = 1 [default = 1.7976931348623157e+308];
  optional string string1 = 2 [default = ""];                   
}

encoding and decoding { string1: "" } gets me

{
  number1: 1,
  string1: ''
}

Protobuf packages and namespaces

Packages and message namespaces are flattened when reading. This makes it impossible to parse a fully qualified path that is functional in other language bindings. For example:

syntax = "proto3";
package Foo.Enums;

enum E {
    A = 0;
    B = 1;
}
syntax = "proto3";
package Foo.Bar;

message Envelope {
    .Foo.Enums.E enum = 1;
}

Cross reference: mapbox/pbf#58

Parser tokenizes reserved characters inside of strings

Let's say I have the following protocol buffer field:

repeated Basic list = 1 [packed='some,list,value'];

Or one with brackets:

repeated Basic list = 1 [packed="[some], [lists], [here]"];

This is valid protobuf, but the parser will attempt to tokenize those reserved characters inside the string when it should instead ignore them and push the entire string as a token. This causes the parser to fail.

I implemented a solution that looks for quotes ' or " and then performs a lookahead to find closing quotes and pushing the entire string to options field.

I'd be happy to submit a PR for this if there is no more elegant way to fix this without rewriting much of the parser.

[enhancement] Ensure that tag numbers have been assigned and throw an error if not

Forgetting to assign a tag number to a property leads to subtle bugs. In the below example dollars will round trip fine on an encode/decode, but pesos will come out null.
It'd be nice if the parser could ensure that tags were assigned and throw an error if not to make it easier to see the problem.

message wallet {
  optional int32 dollars;
  optional int32 pesos;
}

message traveller {
  required wallet wallet;
}

Implement Protobuf Extensions

Hi,
we would also need to use functionality "Protobuf Extensions" (https://developers.google.com/protocol-buffers/docs/proto#extensions). I've already implemented this in the protobuf-schema repo bydga@ff72a17.

Could you please guide me, if its also necessary to edit the protocol-buffer repo - and if where should i start looking? I mean the binary serialization in files like https://github.com/mafintosh/protocol-buffers/blob/master/encodings.js? Or would it get integrated "somehow automatically"? :).

Support (ignore?) custom options

I'm using https://github.com/devongovett/protobuf-jsonschema which depends on this project to map protocol buffer IDLs to JSON schema. I also use https://github.com/GoogleCloudPlatform/protoc-gen-bq-schema/ to generate a BigQuery schema.

This last part requires providing a table name via a custom option named gen_bq_schema.table_name (see https://developers.google.com/protocol-buffers/docs/proto#options), for example

package foo;
import "bq_table_name.proto";

message Bar {
  option (gen_bq_schema.table_name) = "bar_table";

  message Nested {
    repeated int32 a = 1;
  }

  required int32 a = 1;
  optional Nested b = 2;
  repeated string c = 3;
}

At the moment, this fails:

/usr/local/lib/node_modules/protobuf-jsonschema/node_modules/protocol-buffers-schema/parse.js:90
        throw new Error('Unexpected token in message field: ' + tokens[0])
        ^

Error: Unexpected token in message field: en_bq_schema.table_name
    at onfield (/usr/local/lib/node_modules/protobuf-jsonschema/node_modules/protocol-buffers-schema/parse.js:90:15)
    at onmessagebody (/usr/local/lib/node_modules/protobuf-jsonschema/node_modules/protocol-buffers-schema/parse.js:161:26)
    at onmessage (/usr/local/lib/node_modules/protobuf-jsonschema/node_modules/protocol-buffers-schema/parse.js:210:14)
    at parse (/usr/local/lib/node_modules/protobuf-jsonschema/node_modules/protocol-buffers-schema/parse.js:583:30)
    at Compiler.open (/usr/local/lib/node_modules/protobuf-jsonschema/index.js:17:16)
    at new Compiler (/usr/local/lib/node_modules/protobuf-jsonschema/index.js:9:22)
    at module.exports (/usr/local/lib/node_modules/protobuf-jsonschema/index.js:192:18)
    at Command.<anonymous> (/usr/local/lib/node_modules/protobuf-jsonschema/cli.js:13:18)
    at Command.listener (/usr/local/lib/node_modules/protobuf-jsonschema/node_modules/commander/index.js:301:8)
    at emitOne (events.js:96:13)

As far as I can tell, this could safely be ignored for now, just like the reservedkeyword was yesterday

my gen error, because have parenthesis。

because doc have parenthesis!and parse error。
can you help me reolve it! thanks very much.

.proto file like this:

	rpc AdminRecommendProductImport(msg) returns (msgResp) {
		option (common.auth).name = true;
	}

throw new Error('Expected = but found ')

message MessageEntityTest {
enum Event {
reserved 1,4;
BACKGROUND = 2;
FOREGROUND = 3;
}
optional Event event = 1;
}

when parsed the symbol of 'reserved', will throw error!

Add imports to stringify method

I'd like to be able to use the library as part of a protobuf file generation project I am working on but am running into an issue where imports are not preserved when calling parse to transform protobuf to JSON and then calling stringify to stringify the JSON back into the .proto format.

It seems like a solution to this would be pretty simple. Happy to create a PR if desired.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.