nytimes / openapi2proto Goto Github PK

View Code? Open in Web Editor NEW

942.0 942.0 100.0 692 KB

A tool for generating Protobuf v3 schemas and gRPC service definitions from OpenAPI specifications

License: Apache License 2.0

Go 100.00%

openapi2proto's People

Contributors

Stargazers

Watchers

Forkers

danny-cassidy digideskio tmc gaplyk karux voxadam krodyx gophersgang pboyer runeksvendsen sarahkevinking ctborg sgunadhya jkinkead roscopecoltran omb amitkumarj441 cbroglie hilalisa boomerchi chrusty bienne kaelzhang81 infotech-group karhoo dnfehren aaaasssdd lestrrat davepuchyr hoardboard sysradium renesugar robertjchristian ravindranathakila dvbnrg openbnb andradeandrey vrdate sauganth nickman loudent2 bictorman sukidesuaoi mo1seev seeknode gmwxio 2blackcoffees dio franky-li datphan310 davidsansome onimsha cesarblum divyabasu elyashivlavi finderseyes darrenmcc blaizedsouza lianasianturi zhutony izzunnaqi agasparovic-sabre obitel-forks magnacerta luigi-riefolo valery-barysok curiousleo jasonfriedland kalixi lunarforge b25 isgasho rizalgowandy henrywu2019 a-pedroso hkraftno michaelandrepearce mhd999 chriscitrus hexfusion unionj-cloud go-language-server truyet xincao9 maurik77 davidvmar devpaltestaccount bajalnyt valcharry iq-scm sanposhiho izouxv sshehrozali shirenren ayazkhuraishi alainlompo shixulu kms9 alexanderstr2

openapi2proto's Issues

What should happen when there are multiple `in: body` parameters?

What should this generate for the http annotation? (abbreviated)

parameters:
   - name: foo
      in: body
   - name: bar
      in: body
   - name: baz
      in: body

Currently this just takes the first parameter and does

rpc .... {
    option (google.api.http) = {
      body: "foo"
    };
}

But this probably isn't right. Reading this I'm inclined to just assume everything is in the body (i.e. body: *)?

Force generated enums to work with C++

Add parent message names as a prefix to avoid collisions
Force enum values to uppercase to follow conventions
Add this to list of caveats

Title with '&' character results in a package name with an '&' character.

Really great tool. Thank you so much for this.

A json swagger file with

"title": "API to View & Search for Items",

Will result in a proto with

package apitoview&searchforitems;

openapi2proto should probably take special characters into account?

Lower-case references in responses aren't capitalized

A response definition like:

"responses": { 
    "200": { 
        "description": "Successful response",
        "schema": { 
            "$ref": "#/definitions/status"
        } 
    },
}

Generates a proto message Status, but an RPC definition returning status.

Fix in a moment.

Add support for protobuf custom options

Protobuf supports creation of custom options but unfortunately the openApi2Proto don't support this.
It would be great if you could use the openapi extension pattern to define custom options, maybe something like x-option-name: value

Extensions in OpenApi
https://github.com/OAI/OpenAPI-Specification/blob/master/versions/3.0.0.md#specificationExtensions

CustomOptions in protobuf
https://developers.google.com/protocol-buffers/docs/proto#customoptions

Include http annotations for grpc-gateway users

https://github.com/googleapis/googleapis/blob/master/google/api/http.proto

https://github.com/gengo/grpc-gateway

The end result should look something like:

  rpc Echo(StringMessage) returns (StringMessage) {
    option (google.api.http) = {
      post: "/v1/example/echo"
      body: "*"
    };
  }

Array Enum Definition

I'm currently encountering an issue involving a definition containing an array of string enum. Here's my swagger.yaml:

swagger: '2.0'
info:
  title: Example
  version: "1.0.0"
paths:
  /endpoint/{id}:
    get:
      tags:
        - ExampleTag
      parameters:
        - name: id
          in: path
          required: true
          type: string
      responses:
        '200':
          description: Example description
          schema:
            type: array
            items:
              $ref: '#/definitions/FirstDefinition'


definitions:
  FirstDefinition:
    type: object
    properties:
      first_property:
        type: array
        items:
          type: string
          enum:
            - A
            - B

Here's what I get:

syntax = "proto3";

import "google/api/annotations.proto";

package example;

message GetEndpointIdRequest {
    string id = 1;
}

message GetEndpointIdResponse {
    repeated FirstDefinition items = 1;
}

message FirstDefinition {
    repeated string first_property = 1;
}

service ExampleService {
    rpc GetEndpointId(GetEndpointIdRequest) returns (GetEndpointIdResponse) {
      option (google.api.http) = {
        get: "/endpoint/{id}"
      };
    }
}

I'm kind of expecting my enum ["A","B"] to be present somewhere.

Add support for Open API spec v3.0.0

Helpful walkthrough: https://dev.to/mikeralphson/comparing-openapiswagger-20-and-300-rc1

Any plans to proto2openapi?

Fix formatting of nested message types

Ever since the update that added comments, nested messages are missing an indent

Allow relative and remote schema references in $ref

https://github.com/OAI/OpenAPI-Specification/blob/master/guidelines/REUSE.md#guidelines-for-referencing

don't know how to handle schema type '[boolean]'

Attempted to generate proto from the PagerDuty swagger spec. Got this error:

error: failed to transpile: failed to compile OpenAPI spec to Protocol buffers: failed to compile parameters: failed to compile #/parameters/schedule_overflow: don't know how to handle schema type '[boolean]'

Simple proposed fix forthcoming.

fails on example swagger document

https://apis-guru.github.io/api-models/github.com/v3/swagger.yaml

invalid $ref object referenced with a type of %!s(<nil>)

I am exporting my spec from SwaggerHub and am getting this error at start:

invalid $ref object referenced with a type of %!s(<nil>)

My API is sensitive, but I can share it privately if necessary.

Thanks

Consider using a different diff library

I wrote a few PRs so far, and I must say the diff library being used in the test is not quite up to par: For example, it couldn't detect a difference between a payload with and without a trailing comma. And in other situations I got really confusing outputs, so I had to rely on priting t.Logf("%x", ...) to see the hex output and do a diff-by-your-own-eyes.

Would it possible to consider using github.com/pmezard/go-difflib? It seems to be the least broken library out there.

Use go-openapi library for parsing?

The go-openapi group has very rich Swagger support, particularly for parsing & resolving references.

It might make sense to rewrite the generation logic to use this library, instead of re-writing the swagger logic from scratch.

I might take a stab at this in the future, if this sounds reasonable.

Externally defined objects

I tried to use externally defined types with openapi2proto but could not make it work nicely.

I found a PR which makes the parsers aware of the external paths in $ref however it produces weird results.

For example, if you define a type with:

          schema:
            $ref: 'Pet.yaml'

It creates a service with Pet.yaml:

rpc PostPets(PostPetsRequest) returns (Pet.yaml) {}

At the same time in the object you use that ref in it replaces the filepath with the package name + message name as I expect:

message GetPetsResponse {
    repeated pet.Pet items = 1;
}

Besides that I see no import pets.proto added and lastly - who is responsible for generating the pets.proto file? It seems impossible by design for openapi2proto to produce it as it spits the generated proto to the stdout, which presumes you to have just one final file.

I tried using openapi2proto with the particular Pey.yaml file. It produced just an empty proto file.
So can you explain how can we split swagger API spec and models definitions and still be able to use this tool?

PS. The Pey example I've used is located here https://github.com/OAI/OpenAPI-Specification/tree/master/examples/v2.0/yaml/petstore-separate/spec

Add support for parameters with `collectionFormat: csv`

Hi,

this works:

      parameters:
        - name: id
          in: path
          description: ID of pet to use
          required: true
          type: array
          items:
            type: string
          collectionFormat: csv

-> repeated string id = 3;

Does not work:

      parameters:
        - $ref: "#/parameters/id"
...
parameters:
  id:
    in: path
    name: id
    description: ID of pet to use
    required: true
    type: array
    items:
      type: string
    collectionFormat: csv

array id = 1;
...
repeated int32 id = 1

Do you have any idea?

Rgds
Tobi

Wrappers for map values of type list are generated multiple times

For example:

// automatically generated wrapper for a list of Discount items
message DiscountsList {
    repeated Discount discounts = 1;
}

// automatically generated wrapper for a list of Discount items
message DiscountsList {
    repeated Discount discounts = 1;
}

A bug with body arrays in put requests?

When I have openAPI yaml that looks like:

    put:  
      summary: Customer data
      description: |
        Accepts the definitions of a set of customers.
      parameters:
        - name: numberOfCustomers
          in: query
          description: Number of customers being posted.
          required: true
          type: integer
          format: int32
        - name: customerData
          in: body
          description: Array of customer data.
          required: true
          schema:
            type: array
            items:
              $ref: '#/definitions/Customer'
      responses:
   [snip]

I get .proto code that looks like

message PutCustomerRequest {
    // Array of customer data.
    ;
    // Number of customers being posted.
    int32 numberOfCustomers = 2;
}

The customer object is properly defined and generated a perfectly fine declaration itself

message Customer {
    StreetAddress address = 1;
    // Unique identifier for the customer.
    int32 customerId = 2;
    // Name of customer.
    string customerName = 3;
}

Is this a bug, or am I asking it to do something I shouldn't ask it to do?

Thanks!

Create a translation table for types

We should have some visual way of representing what Open API types and formats translate into for protobuf along with which bits of Open API do not get included in the proto definition.

Examples:
https://developers.google.com/protocol-buffers/docs/proto3#scalar
https://github.com/OAI/OpenAPI-Specification/blob/master/versions/2.0.md#data-types

Add support for `enum` types in "Definitions"

panic: interface conversion: interface {} is nil, not string

I'm fetching this swagger spec file: https://esi.tech.ccp.is/latest/swagger.json?datasource=tranquility

When I run openapi2proto against it I get an error (and massive stacktrace). Not sure what's going on as that's not completely descriptive to me.

11:24 $ openapi2proto -spec esi.yaml
panic: interface conversion: interface {} is nil, not string [recovered]
	panic: interface conversion: interface {} is nil, not string [recovered]
	panic: interface conversion: interface {} is nil, not string

goroutine 1 [running]:
text/template.errRecover(0xc420095ce8)
	/usr/lib/go-1.8/src/text/template/exec.go:140 +0x1ec
panic(0x72a7c0, 0xc420307dc0)
	/usr/lib/go-1.8/src/runtime/panic.go:489 +0x2cf
text/template.errRecover(0xc4200949d8)
	/usr/lib/go-1.8/src/text/template/exec.go:140 +0x1ec
panic(0x72a7c0, 0xc420307dc0)
	/usr/lib/go-1.8/src/runtime/panic.go:489 +0x2cf
github.com/NYTimes/openapi2proto.(*Items).ProtoMessage(0xc42024a000, 0xc4202cf890, 0x30, 0xc420242a40, 0x8, 0xc420406180, 0xc42028c5f0, 0x0, 0x0, 0x0)
	/home/wonko/go/src/github.com/NYTimes/openapi2proto/openapi.go:249 +0x12b9
reflect.Value.call(0x7379e0, 0xc42024a000, 0xa13, 0x77ddae, 0x4, 0xc4204cb800, 0x5, 0x5, 0xc400009514, 0x6e2514, ...)
	/usr/lib/go-1.8/src/reflect/value.go:434 +0x91f
reflect.Value.Call(0x7379e0, 0xc42024a000, 0xa13, 0xc4204cb800, 0x5, 0x5, 0x8e48a0, 0xc420011b30, 0x70c480)
	/usr/lib/go-1.8/src/reflect/value.go:302 +0xa4
text/template.(*state).evalCall(0xc420094958, 0x7379e0, 0xc42024a000, 0x16, 0x7379e0, 0xc42024a000, 0xa13, 0x8e48a0, 0xc42013ab70, 0xc42013c1e6, ...)
	/usr/lib/go-1.8/src/text/template/exec.go:668 +0x51f
text/template.(*state).evalField(0xc420094958, 0x7379e0, 0xc42024a000, 0x16, 0xc42013c1e6, 0xc, 0x8e48a0, 0xc42013ab70, 0xc42006a580, 0x6, ...)
	/usr/lib/go-1.8/src/text/template/exec.go:558 +0xe18
text/template.(*state).evalFieldChain(0xc420094958, 0x7379e0, 0xc42024a000, 0x16, 0x7379e0, 0xc42024a000, 0x16, 0x8e48a0, 0xc42013ab70, 0xc42013c210, ...)
	/usr/lib/go-1.8/src/text/template/exec.go:526 +0x22b
text/template.(*state).evalVariableNode(0xc420094958, 0x7379e0, 0xc42024a000, 0x16, 0xc42013ab70, 0xc42006a580, 0x6, 0x8, 0x0, 0x0, ...)
	/usr/lib/go-1.8/src/text/template/exec.go:514 +0x1ac
text/template.(*state).evalCommand(0xc420094958, 0x7379e0, 0xc42024a000, 0x16, 0xc42013ab10, 0x0, 0x0, 0x0, 0xc4200944b0, 0xc420094538, ...)
	/usr/lib/go-1.8/src/text/template/exec.go:438 +0x900
text/template.(*state).evalPipeline(0xc420094958, 0x7379e0, 0xc42024a000, 0x16, 0xc4201403c0, 0x0, 0x0, 0x0)
	/usr/lib/go-1.8/src/text/template/exec.go:406 +0xf2
text/template.(*state).walk(0xc420094958, 0x7379e0, 0xc42024a000, 0x16, 0x8e42a0, 0xc420011b60)
	/usr/lib/go-1.8/src/text/template/exec.go:232 +0x55a
text/template.(*state).walk(0xc420094958, 0x7379e0, 0xc42024a000, 0x16, 0x8e45a0, 0xc420011830)
	/usr/lib/go-1.8/src/text/template/exec.go:240 +0x139
text/template.(*state).walkRange.func1(0x70ccc0, 0xc42028ca10, 0x98, 0x7379e0, 0xc42024a000, 0x16)
	/usr/lib/go-1.8/src/text/template/exec.go:327 +0x136
text/template.(*state).walkRange(0xc420094958, 0x744740, 0xc42028c5d0, 0x99, 0xc420018740)
	/usr/lib/go-1.8/src/text/template/exec.go:344 +0x5a2
text/template.(*state).walk(0xc420094958, 0x744740, 0xc42028c5d0, 0x99, 0x8e4720, 0xc420018740)
	/usr/lib/go-1.8/src/text/template/exec.go:243 +0x4f1
text/template.(*state).walk(0xc420094958, 0x744740, 0xc42028c5d0, 0x99, 0x8e45a0, 0xc420144720)
	/usr/lib/go-1.8/src/text/template/exec.go:240 +0x139
text/template.(*Template).execute(0xc4201561c0, 0x8deac0, 0xc42020f570, 0x744740, 0xc42028c5d0, 0x0, 0x0)
	/usr/lib/go-1.8/src/text/template/exec.go:195 +0x20a
text/template.(*Template).Execute(0xc4201561c0, 0x8deac0, 0xc42020f570, 0x744740, 0xc42028c5d0, 0xc4202cf890, 0x30)
	/usr/lib/go-1.8/src/text/template/exec.go:178 +0x53
github.com/NYTimes/openapi2proto.Parameters.ProtoMessage(0xc420140730, 0x7, 0x9, 0x0, 0x0, 0x0, 0xc4202cf800, 0x29, 0xc420406180, 0x6cf4ef, ...)
	/home/wonko/go/src/github.com/NYTimes/openapi2proto/openapi.go:699 +0x1fc
github.com/NYTimes/openapi2proto.(*Endpoint).protoMessages(0xc42015c7e0, 0x0, 0x0, 0x0, 0xc4202cf800, 0x29, 0xc420406180, 0xc420228600, 0x5a7)
	/home/wonko/go/src/github.com/NYTimes/openapi2proto/openapi.go:573 +0xc1
github.com/NYTimes/openapi2proto.(*Path).ProtoMessages(0xc42013f3c0, 0xc420240d80, 0x2f, 0xc420406180, 0x0, 0x0)
	/home/wonko/go/src/github.com/NYTimes/openapi2proto/openapi.go:631 +0x375
reflect.Value.call(0x7298c0, 0xc42013f3c0, 0x613, 0x77ddae, 0x4, 0xc4202ce150, 0x2, 0x2, 0xc40000d8f0, 0x6e68f0, ...)
	/usr/lib/go-1.8/src/reflect/value.go:434 +0x91f
reflect.Value.Call(0x7298c0, 0xc42013f3c0, 0x613, 0xc4202ce150, 0x2, 0x2, 0x8e48a0, 0xc420144360, 0x7266c0)
	/usr/lib/go-1.8/src/reflect/value.go:302 +0xa4
text/template.(*state).evalCall(0xc420095c68, 0x7298c0, 0xc42013f3c0, 0x16, 0x7298c0, 0xc42013f3c0, 0x613, 0x8e48a0, 0xc420144300, 0xc42015404a, ...)
	/usr/lib/go-1.8/src/text/template/exec.go:668 +0x51f
text/template.(*state).evalField(0xc420095c68, 0x7298c0, 0xc42013f3c0, 0x16, 0xc42015404a, 0xd, 0x8e48a0, 0xc420144300, 0xc420156080, 0x3, ...)
	/usr/lib/go-1.8/src/text/template/exec.go:558 +0xe18
text/template.(*state).evalFieldChain(0xc420095c68, 0x7298c0, 0xc42013f3c0, 0x16, 0x7298c0, 0xc42013f3c0, 0x16, 0x8e48a0, 0xc420144300, 0xc420154070, ...)
	/usr/lib/go-1.8/src/text/template/exec.go:526 +0x22b
text/template.(*state).evalVariableNode(0xc420095c68, 0x7298c0, 0xc42013f3c0, 0x16, 0xc420144300, 0xc420156080, 0x3, 0x4, 0x0, 0x0, ...)
	/usr/lib/go-1.8/src/text/template/exec.go:514 +0x1ac
text/template.(*state).evalCommand(0xc420095c68, 0x7298c0, 0xc42013f3c0, 0x16, 0xc4201442a0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/usr/lib/go-1.8/src/text/template/exec.go:438 +0x900
text/template.(*state).evalPipeline(0xc420095c68, 0x7298c0, 0xc42013f3c0, 0x16, 0xc4201420a0, 0x0, 0x0, 0xc4200958c8)
	/usr/lib/go-1.8/src/text/template/exec.go:406 +0xf2
text/template.(*state).walk(0xc420095c68, 0x7298c0, 0xc42013f3c0, 0x16, 0x8e42a0, 0xc420144390)
	/usr/lib/go-1.8/src/text/template/exec.go:232 +0x55a
text/template.(*state).walk(0xc420095c68, 0x7298c0, 0xc42013f3c0, 0x16, 0x8e45a0, 0xc420144240)
	/usr/lib/go-1.8/src/text/template/exec.go:240 +0x139
text/template.(*state).walkRange.func1(0x70ccc0, 0xc420136a40, 0x98, 0x7298c0, 0xc42013f3c0, 0x16)
	/usr/lib/go-1.8/src/text/template/exec.go:327 +0x136
text/template.(*state).walkRange(0xc420093c68, 0x742440, 0xc42026ad50, 0x99, 0xc420156100)
	/usr/lib/go-1.8/src/text/template/exec.go:344 +0x5a2
text/template.(*state).walk(0xc420093c68, 0x742440, 0xc42026ad50, 0x99, 0x8e4720, 0xc420156100)
	/usr/lib/go-1.8/src/text/template/exec.go:243 +0x4f1
text/template.(*state).walk(0xc420095c68, 0x742440, 0xc42026ad50, 0x99, 0x8e45a0, 0xc420011500)
	/usr/lib/go-1.8/src/text/template/exec.go:240 +0x139
text/template.(*Template).execute(0xc4200185c0, 0x8deac0, 0xc42020e230, 0x742440, 0xc42026ad50, 0x0, 0x0)
	/usr/lib/go-1.8/src/text/template/exec.go:195 +0x20a
text/template.(*Template).Execute(0xc4200185c0, 0x8deac0, 0xc42020e230, 0x742440, 0xc42026ad50, 0x0, 0xc420136470)
	/usr/lib/go-1.8/src/text/template/exec.go:178 +0x53
github.com/NYTimes/openapi2proto.GenerateProto(0xc42020a000, 0x0, 0xc42020a000, 0x0, 0x0, 0xc420136428, 0xc420136470)
	/home/wonko/go/src/github.com/NYTimes/openapi2proto/proto.go:171 +0x307
main.main()
	/home/wonko/go/src/github.com/NYTimes/openapi2proto/cmd/openapi2proto/main.go:21 +0x1e1

Query strings in paths not gracefully handled.

Query strings are not allowed in swagger paths, but some tooling appears to tolerate their presence.

When a query string is in the path, it gets appended, with its ? character intact, to all path-based names in the proto file. It should be handled gracefully (probably stripped).

Fix in a moment.

Adopt a more flexible way of resolving $ref references

We still have the kubernetes.json fixture commented out from tests due to a (valid) JSON Schema $ref to point to a string type:

     "$ref": {
      "type": "string"
     },

We should be able to detect this and simply resolve it as a type: string.

Hopefully this is the only thing preventing us from using the k8s spec as a fixture :)

There is a panic when having responses with no schemas

This seems to be an accepted use case, response schemas are not required by the OpenAPI spec. I guess generating google.protobuf.Empty may make sense.

Integer Parameter Definition Object

Appears to be an issue converting Parameters Definitions Object with type integer even when format is specified. Can be reproduced by changing SpecificConceptParam in https://github.com/NYTimes/openapi2proto/blob/master/fixtures/semantic_api.json to

   "SpecificConceptParam": {
        "name": "specific-concept",
        "in": "path",
        "description": "The name of the concept, used for Constructing a Semantic API Request by Concept Type and Specific Concept Name. The parameter is defined in the URI path, as the element immediately preceding \".json\" like with \"Baseball.json\".\n",
        "required": true,
        "type": "integer",
        "format": "int32"
}

Currently, the resulting .proto shows "integer" instead of the expected "int32"

   string query = 3;
   integer specific_concept = 4;
}

message names and field names must not collide

During the course of investigating #79, I ran protoc (libprotoc 3.5.1) against one of the fixtures to find out the "correct" protobuf definition to generate, and I found that definitions like this https://github.com/NYTimes/openapi2proto/blob/master/fixtures/accountv1-0.proto#L321-L327 die with a message like

foo.proto:27:17: "Amount" is already defined in "accountinformationapis.AccountBalance.Data".

After a bit of poking around, I think the rule is as follows:

1: field names and message names may not collide in the same level

This seems to be illegal

message Foo {
   message Bar {}
   string Bar = 1;
}

$ protoc --proto_path . --go_out=plugins=grpc:pb bar.proto  
bar.proto:6:11: "Bar" is already defined in "bar.Foo".

If we change either the field name or the message name, this goes away

2: if the field is nested, it's not a collision

message Foo {
  message Bar {}
  message Baz {
    string Bar = 1;
  }
}

$ protoc --proto_path . --go_out=plugins=grpc:pb bar.proto 
(no error)

preserve descriptions and summary fields

https://github.com/tmc/grpc-uber/blob/master/uber.proto this generated file doesn't reflect the comments from https://github.com/tmc/grpc-uber/blob/master/uber.openapi.yaml

Capitalize enum type names

Currently enum type names are not capitalized when creating a nested enum type.

For example, a 'type' enum within a 'Background' message currently gets named 'Background_type'. We should have it output 'Background_Type'.

Check internet connectivity before running HTTP tests

Sometimes I work when there's connectivity, sometimes I don't have connection. It's really annoying for a test to fail just because of this.

Would it be feasible to skip the tests that downloads specs from the internet when appropriate? I can see two possibilities:

a. Skip if there's no connectivity -- that is, if http.Get(url) fails, don't run the subsequent test.
b. Skip if the test if -short is specified. This at least gives you the option to skip if desired.

Cannot unmarshal bool into Go struct field XXX of type openapi2proto.XXXX

Golang version: go version go1.10 darwin/amd64.

Step by step to reproduce:

go get -u github.com/NYTimes/openapi2proto/cmd/openapi2proto
Trying to convert Swagger file from https://github.com/apigee/openbank/blob/master/openapi/accountv1-0.json
openapi2proto -spec accountv1-0.json -options
Failed to convert.

2018/03/02 00:23:30 unable to load spec: unable to parse referenced file: json: cannot unmarshal bool into Go struct field Items.additionalProperties of type openapi2proto.Items

Expectation:
Should not see error and able to convert swagger file.

Add support for 'map' types

We should add the check for additionalProperties in the spec: https://github.com/OAI/OpenAPI-Specification/blob/master/versions/2.0.md#model-with-mapdictionary-properties

...and generate an appropriate map field: https://developers.google.com/protocol-buffers/docs/reference/proto3-spec#map_field

Allow untyped objects to be specified as bytes

In many of our specs, we have things like this:

      authorization_data:
        type: object
        description: Data, as an arbitrary json blob

      authorization_data:
        type: object
        description: Data, as an arbitrary json blob
        additionalProperties: true

This allows arbitrary data, per https://swagger.io/docs/specification/data-models/dictionaries/

Currently, this will be generated as an empty Message or a map<string, empty Message> by this library. This does't work well. My proposal is that these should actually be modeled as the scalar type bytes. That way, whatever language implementation is using the schema can decide how to convert the arbitrary data into and from bytes.

Another option would be to use google.protobuf.Any, as it does basically the same thing, but also allows packing/unpacking to different Message types.

@jprobinson any thoughts on what is preferred?

"number" defaults to integral type; should default to floating-point

Per the Swagger spec (unchanged from 2.0), the number type on a primitive should default to floating-point, not integer.

Fix in a moment.

integer, long number type should be written as int32, int64

    properties:
      value:
        type: number
        format: long
        description: Numerical duration length.
        default: 8

long is a valid swagger type. This should be written as int64 in protobuf format. Currently it is written as long.

Same issue occurs for integer - it should be written as int32.

message names not camel-cased

I thought I fixed this, but

definitions:
   foo_bar:
      ...

produces a message like

message Foo_bar  {
...
}

Inspect $ref's to verify they are object types before referencing as message types.

Sometimes $refs are handy to reuse string formatting. We shouldn't force those references to be message types.

Support data model using allOf

Exemple file

openapi: "3.0.0"
info:
  description: "This is a example swagger file"
  version: "1.0.0"
  title: "Swagger Example"
components:
  schemas:
    Person:
      type: object
      required:
      - name
      properties:
        gender:
            type: string
            description: Gender of the person
            default: Mr
            enum:
            - Miss
            - Mr
            - Mrs
            - Ms
        name:
          type: string
        address:
          $ref: '#/definitions/Address'
        age:
          type: integer
          format: int32
          minimum: 0
        map:
          type: object
          additionalProperties:
            type: string
        hobbies:
          type: array
          items:
            type: string
          example: ["sport", "music", "movies"]
    ExtendedPerson:
      properties:
        allOf:
        - $ref: '#/definitions/Person'
        - type: object
          required:
          - birthday
          properties:
            birthday:
              type: string
              format: date
    Address: 
      type: object 
      properties: 
        streetNumber: 
          type: integer
          format: int32
        streetName: 
          type: string 
        postalCode: 
          type: integer

Give

    syntax = "proto3";

    package ;

    message Address {
        int32 postalCode = 1;
        string streetName = 2;
        int32 streetNumber = 3;
    }

    message ExtendedPerson {}

    message Person {
        enum PersonGender {
            PERSON_GENDER_MISS = 0;
            PERSON_GENDER_MR = 1;
            PERSON_GENDER_MRS = 2;
            PERSON_GENDER_MS = 3;
        }

        Address address = 1;
        int32 age = 2;

        // Gender of the person
        PersonGender gender = 3;
        repeated string hobbies = 4;
        map<string, string> map = 5;
        string name = 6;
    }

message ExtendedPerson {} is not generated

Should return

    message ExtendedPerson {
        int32 postalCode = 1;
        string streetName = 2;
        int32 streetNumber = 3;
        string birthday = 4;
    }

Array as additionalPrperties

Hey All,

My team at Autodesk is looking to use this but we ran into a bug when using arrays as addtionalProperties e.g.

definitions:
  testResponse:
    type: object
    properties:
      testThings:
        type: array
        description: 'list of things'
        items:
          $ref: '#/definitions/TestThings'
      mappedTestThings:
        type: object
        description: 'map of testThingStuff'
        additionalProperties:
          type: array
          description: list of test thing Stuff
          items:
            $ref: '#/definitions/TestThingStuff'

I forked the code and I have what I think is a fix ready. Are you interested in having my changes merged?

cheers,
Troy Louden
Autodesk Cloud Licensing Services

Unable to set x-global-options which have a boolean value eg. java_multiple_files: true

Unable to set x-global-options which are booleans like java_multiple_files: true

Error returned is.
error: failed to transpile: failed to load OpenAPI spec: failed to decode content: json: cannot unmarshal bool into Go struct field Spec.x-global-options of type string

example from top of swagger yaml package name works fine.

x-global-options:
  java_package: com.mypackagename
  java_multiple_files: true # THIS CAUSES THE ERROR

Use extension field `x-proto-tag` to allow users to pre-declare Protobuf tags

One of the biggest hurdles of this tool is the inability to easily handle add new attributes to an existing schema as the Protobuf tags generated are based on properties alphabetical order.

To solve this, we should lean on Open API extensions and a new, optional attribute "x-proto-tag` for allowing users to preset the attribute tag values.

Allow the use of the `parameters` object in the base schema

https://github.com/OAI/OpenAPI-Specification/blob/master/versions/2.0.md#schema

The parameters block should be treated similar to the definitions block.

require to specify the attributes as optional if not required in the spec

Add command line option to tell protobuf compiler to generate multiple java files

This would be added to the generated .proto file at the top, below package:

option java_multiple_files = true;

It looks like by adding this:

x-global-options:
  java_multiple_files: true

to the spec, the schema won't compile because it expects the option value to be a string. Some protobuf options, like java_multiple_files, are boolean.

Add protobuf options

I would like to add arbitrary protobuf options to the generated file, namely go_package

option go_package = "myawesomepackage"

I can see two ways to achieve this:

1. Add an option to the command line.

openapi2proto -new_flag go_package=myawesomepackage ...

where new_flag would be replaced with the appropriate name (I was inclined to use option, but it was already taken)

2. Add an custom/extra field to the spec

I'm not an OpenAPI spec expert, but this being a rather free form text file, I'm sure there's room to add extra fields. Here I assume it's okay for us to add a top level field name x-global-options, but it could be anywhere

openapi: 2.0
x-global-options:  # made this up
   go_package: myawesomepackage
...

I'd be willing to write up a patch if we can decide on a direction to go.

Snake to Camel Case

Currently, the library doesn't do much work on the path name received from swagger. As an example, it converts POST: /queue/{id}/enqueue_player to something like PostQueuesIdEnqueue_player. I think I'd like that to be PostQueuesIdEnqueueGroup, which seems like a good default for the library. Mind if I send a pull request for this?

Completely Broken: findRefByName breaks conversion

Great library by the way thank you for making it!

While using it today I ran across an interesting bug

func findRefName(i *Items, defs map[string]*Items) string {
	if i.Name != "" {
		return i.Name
	}

	itemType := strings.TrimLeft(i.Ref, "#/parameters/")
	item, ok := defs[itemType]

	if !ok {
		log.Fatalf("unable to find referenced type for parameter: %#v", i)
	}

	return item.Name
}

TrimLeft removes all characters from i.Ref that are in the set of characters #/parmets. So any parameter such as #/parameters/access_token becomes ccess_token because a is part of the set of characters that we are trimming off. The fix is to use TrimPrefix instead of TrimLeft. PR incoming.

RFC: change how protobuf is generated

In #75 and #78, I found discrepancies on the names generated by the tool for type names that should actually point to the same protobuf message.

The root cause for this is that there is no central authority in the code that can resolve OpenAPI names to protobuf types -- the code is basically converting the OpenAPI names to a type name on the fly, and if happen to not to use the same converter function in multiple places, we're doomed.

I can certainly keep on sending PRs, but at the end, this just keeps a gaping possibility for a subtle failures, and I don't think I like to build my infrastructure if at least debugging this is easier.

That's the lead up to my suggestion: having spent enough time fixing bugs in the past few days, I think I would really like to see this tool to be refactored from the ground up so it's easier to manipulate things when the need arises in the future.

I'd like to propose to break this tool up into three components:

openapi loader
openapi-to-protobuf converter
protobuf encoder

The loader will read an openapi spec from a source, and apply any "fixups". The converter will take the openapi data structure and convert it to a protobuf structure. and finally, the encoder will traverse the protobuf structure and print out the protobuf definitions.

The secret sauce will be in the converter, and this is where we will be keeping a type of system of sorts, so we can fix the problem of resolving an openapi type name to its equivalent in protobuf.

Thoughts?

Support for proto to swagger ?

The tool looks great! Thanks for sharing this. Are there any plans to somehow support the reverse ( i.e. proto3 to swagger or json schema ? )

nytimes / openapi2proto Goto Github PK

openapi2proto's People

Contributors

Stargazers

Watchers

Forkers

openapi2proto's Issues

1: field names and message names may not collide in the same level

2: if the field is nested, it's not a collision

Exemple file

1. Add an option to the command line.

2. Add an custom/extra field to the spec

Recommend Projects

Recommend Topics

Recommend Org