When using a field get function, if the field is a child of a choice field the choice value must be set as part of the field get function.

The following should be generated for field get functions that are a child of a choice field

// JA sets choice to J_A and returns EObject
func (obj *jObject) JA() EObject {
	if obj.obj.JA == nil {
		obj.obj.JA = &sanity.EObject{}
	}
	obj.SetChoice(JObjectChoice.J_A)
	return &eObject{obj: obj.obj.JA}
}

// SetFA sets choice to F_A and returns FObject
func (obj *fObject) SetFA(value string) FObject {
	obj.SetChoice(FObjectChoice.F_A)
	obj.obj.FA = &value
	return obj
}

A test to set a value whose significance is dependent on the sibling choice value

func TestChoice(t *testing.T) {
	api := openapiart.NewApi()
	config := api.NewPrefixConfig()

	f := config.F()
	f.SetFA("a fa string")
	assert.Equal(t, f.Choice(), openapiart.FObjectChoice.F_A)

	j := config.J().Add()
	j.JA().SetEA(22.2)
	assert.Equal(t, j.Choice(), openapiart.JObjectChoice.J_A)

	fmt.Println(config.ToYaml())
}

It's buggy.

  - name: Get all changes
    id: file_changes
    uses: trilom/[email protected]

Discussed in open-traffic-generator/models#139

This issue is noticed while generating in my dev environment:
Shown to @Rangababu-R

Installed dependencies:
go get google.golang.org/protobuf/cmd/protoc-gen-go
go get google.golang.org/grpc/cmd/protoc-gen-go-grpc
go get golang.org/x/tools/cmd/goimports

Path to Go binary:
:/usr/local/go/bin:/home//go/bin

Error logs:
Generating python grpc stubs: /home/padhikary/workspace/athena/openapiart/.env/bin/python -m grpc_tools.protoc --python_out=/home/padhikary/workspace/athena/openapiart/art/sanity --grpc_python_out=/home/padhikary/workspace/athena/openapiart/art/sanity --proto_path=/home/padhikary/workspace/athena/openapiart/art sanity.proto
Generating go stubs: protoc --go_opt=paths=source_relative --go-grpc_opt=paths=source_relative --go_out=/home/padhikary/workspace/athena/openapiart/pkg/sanity --go-grpc_out=/home/padhikary/workspace/athena/openapiart/pkg/sanity --proto_path=/home/padhikary/workspace/athena/openapiart/art --experimental_allow_proto3_optional sanity.proto
protoc-gen-go: program not found or is not executable
Please specify a program using absolute path or make sure the program is available in your PATH system variable
--go_out: protoc-gen-go: Plugin failed with status code 1.
Traceback (most recent call last):
File "artifacts.py", line 41, in
create_openapi_artifacts(openapiart_class)
File "artifacts.py", line 22, in create_openapi_artifacts
openapiart_class(
File "/home/padhikary/workspace/athena/openapiart/openapiart/openapiart.py", line 53, in init
self._generate()
File "/home/padhikary/workspace/athena/openapiart/openapiart/openapiart.py", line 154, in _generate
subprocess.check_call(process_args, shell=False)
File "/usr/lib/python3.8/subprocess.py", line 364, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['protoc', '--go_opt=paths=source_relative', '--go-grpc_opt=paths=source_relative', '--go_out=/home/padhikary/workspace/athena/openapiart/pkg/sanity', '--go-grpc_out=/home/padhikary/workspace/athena/openapiart/pkg/sanity', '--proto_path=/home/padhikary/workspace/athena/openapiart/art', '--experimental_allow_proto3_optional', 'sanity.proto']' returned non-zero exit status 1.

The common.go and openapiartgo.py generator needs to be extended to provide underlying private http functionality in the generated go sdk file.

The Api interface in common.go exposes the following two transport functions which allow transport to be configured using a fluent interface and is not required in the call of NewApi()

type Api interface {
	NewGrpcTransport() GrpcTransport
	NewHttpTransport() HttpTransport
}

Sample test cases for using either grpc or http transport would be the following:

func TestGrpcTransport(t *testing.T) {
	location := "127.0.0.1:5050"
	timeout := 10
	api := NewApi()
	transport := api.NewGrpcTransport().SetLocation("127.0.0.1:5050").SetRequestTimeout(10)
	assert.NotNil(t, transport)
	assert.NotNil(t, transport.Location(), location)
	assert.NotNil(t, transport.RequestTimeout(), timeout)
}

func TestHttpTransport(t *testing.T) {
	location := "https://127.0.0.1:5050"
	verify := false
	api := NewApi()
	transport := api.NewHttpTransport().SetLocation(location).SetVerify(verify)
	assert.NotNil(t, transport)
	assert.NotNil(t, transport.Location(), location)
	assert.NotNil(t, transport.Verify(), verify)
}

should be

        values:
          type: array
          items:
            type: string
          default:
          - 0.0.0.0
          format: ipv4

but currently generated

        values:
          type: array
          items:
            type: string
            default:
            - 0.0.0.0
            format: ipv4

Is your feature request related to a use case? Please describe.

As a user I've created a go ux sdk object and want to be notified of any errors prior to submitting the object to the server. I also don't want to have to write if err != nil on every Set<fieldname> line of code.

The OpenapiArt generator MUST generate a common validation scheme so that when a message is sent over a transport (grpc/http) there are no validation errors present in the message.

Validation errors are those fields that do not fall within the bounds of the supported OpenApi keywords of

• format (supported formats are mac, ipv4, ipv6, hex)
• minimum (may be present when type is number, integer)
• maximum (may be present type is number, integer)
• minItems (may be present when type is array)
• maxItems (may be present when type is array)

Describe the solution you'd like

Two methods of determining if objects are valid.
One can be called explicitly by the user (this allows the same sdk to be used server side) while the other is called implictly by the rpc function.

The following example contains:

• a schema
• go ux sdk generated validation code
• test case demonstrating the two methods of validation

Sample OpenApi schema

components:
  schemas:      
    Sample:
      properties:
        mac_address:
          type: string
          format: mac
        ipv4_address:
          type: string
          format: ipv4
        min_max_value:
          type: integer
          minimum: 6
          maximum: 20
        min_max_list:
          type: array
          minItems: 8
          maxItems: 8
          items:
            type: integer

Sample test case demonstrating how compile time type check passes and run time validation check.

api := NewApi()
sample := api.NewSample()

// the following would all pass compile time check
// any runtime validation that does not pass will be added to an 
// internal struct to be checked on any rpc call or by a Validate
// function that can be called at any time
sample.SetMacAddress("3434334") 
sample.SetIpvAddress("asdfasdf") 
sample.SetMinMaxValue(34343)
sample.SetMinMaxList([]int{1, 2, 3})

// validation check 1
// can be done by a user at any time 
// and the returned error would report the contents from
// all the Set<fieldname> functions above
if err := api.Validate(sample); err != nil {
    panic(err)
}

// validation check 2
// if no validation has been done
// it will be done automatically in the rpc function 
// and the returned error would report the contents from 
// all the Set<fieldname> functions above
// nothing will be sent over the transport
warnings, err := api.SetSample(sample)
if err != nil {
    panic(err)
}

Describe alternatives you've considered

Have each Set<fieldname> function panic as the code is executed. This would require a user to continually follow a practice of code/run/code/run... to resolve validation errors when they can all be reported at the time of the rpc call or an explicit Validate function.

Need to add debug prints based on a flag to allow debugging issues with spec faster.

@SuryyaKrJana FYI

As output of it, let 's identify following:

• What are the items we want to have in Go SDK in coming one week
• What are the items that are there but are buggy
• What are the items that are confusing

@ajbalogh FYI

Make sure the coverage crosses 95% with cov report

Generate go sdk ux file that wraps all protoc generated code with a fluent interface.

protoc generated items to wrap:

• structs with lowercase structs, uppercase interfaces,
• uppercase interface funcs
• fields with funcs: Set
• enums
  • generate string based enums on specific structs per openapi
• choice
  • hide choice and set on child Set func

custom template go code for Api and transports

SetConfig response is always error in go sdk generated file
Responses need to be generated.
Responses need to be added to return values for generated rpc methods.

Currently generated protobuf is not tested.
Neither do we extensively test more detailed spec like OTG openapi.yaml.

Need to understand and implement unit tests for generated spec and SDK (for both protobuf and openapi).

Traceback (most recent call last):
File "C:\Users\rangabar.KEYSIGHT\Desktop\snappi_env\openapiart\lib\site-packages\openapiart\openapiart.py", line 102, in _generate
protobuf.generate(self._openapi)
File "C:\Users\rangabar.KEYSIGHT\Desktop\snappi_env\openapiart\lib\site-packages\openapiart\openapiartprotobuf.py", line 24, in generate
self._write_request_msg(path_object)
File "C:\Users\rangabar.KEYSIGHT\Desktop\snappi_env\openapiart\lib\site-packages\openapiart\openapiartprotobuf.py", line 44, in _write_request_msg
operation = self._get_operation(path_item_object)
File "C:\Users\rangabar.KEYSIGHT\Desktop\snappi_env\openapiart\lib\site-packages\openapiart\openapiartprotobuf.py", line 30, in _get_operation
operation_id = path_item_object['operationId']
TypeError: string indices must be integers
Bypassed creation of protobuf file: string indices must be integers

Add support to remove an item by index

Usage

cfg.ports.add(name='p1')
cfg.ports.add(name='p2')
cfg.ports.add(name='p3')
cfg.ports.remove(1)
assert len(cfg.ports) == 2
assert cfg.ports[0] == 'p1'
assert cfg.ports[1] == 'p3'

Proposed change

class OpenApiIter(OpenApiBase):
    ...
    def remove(self, index):
        del self._items[index]
        self._index = len(self._items) - 1
    ...

It would be good to have support for enum defined at components/schemas level and used in data models by $ref

Global Enum model

Files.FileLocation:
 type: string
 enum:
   - deleted
   - location1
   - location2

Files.FileUploadReply:
 type: object
 properties:
   location:
     $ref: '#/components/schemas/Files.FileLocation'
   filename:
     type: string

Generated Python

class FilesFileLocation(OpenApiObject):
    DELETED = "deleted" # type: str
    LOCATION1 = "location1" # type: str
    LOCATION2 = "location2" # type: str
    
class FilesFileUploadReply(OpenApiObject):
    _TYPES = {
        "location": {
            "type": str,
            "enum": [
                "deleted"
                "location1",
                "location2"
            ],
        }
    }

    @property
    def location(self):
        # type: -> Union[Listeral["deleted"],Literal["location1"],Literal["location2"]]
        return self._get_property("location")

    @location.setter
    def location(self, value):
        # type: (Union[Listeral["deleted"],Literal["location1"],Literal["location2"]])
        self._set_property("location", value)

Sample usage

upload_reply = FilesFileUploadReply()
upload_reply.location = FilesFileLocation.LOCATION1
assert upload_reply.location == FilesFileLocation.LOCATION1

Seeing issues with python2 when generating artifacts.
https://github.com/open-traffic-generator/snappi-convergence/runs/2841872322?check_suite_focus=true#step:5:8

The extension numbers 50001 to 50004 used in OTG are in collision with uhd (uses same numbers) and results in

2021/07/15 15:20:34 WARNING: proto: extension number 50001 is already registered on message google.protobuf.MessageOptions
.........
........
A future release will panic on registration conflicts. See:
https://developers.google.com/protocol-buffers/docs/reference/go/faq#namespace-conflict

Maybe we should switch to a different slot (60000/70000) ?

Refer https://github.com/open-traffic-generator/snappi/issues/81 as an example.

The validation error should point out following in error stack:

• Name of the property and parent property
• Provided type vs expected type
• Provided value

It will better to add another user input say "extension_prefix". So generate artifacts should looks like:

openapiart.OpenApiArt(
    api_files=[
        './tests/api/api.yaml'
        './tests/api/info.yaml'
        './tests/common/common.yaml'
        ], 
    python_module_name='snappi', 
    protobuf_file_name='snappi',
    protobuf_package_name='snappi',
    output_dir='./artifacts',
    extension_prefix='snappi'
)

Therefore final import within api should looks like if user specify "extension_prefix"

lib = importlib.import_module("<extension_prefix>_{}.<python_module_name>".format(ext))

otherwise

lib = importlib.import_module("{}_{}".format(__name__, ext))

I have created one PR for this #17

From @PrasenjitAdhikary

During protobuf deserialization at gRPC server end, we are facing issue with enum fields, if enum field value is set to a value matching with zero.

Example, if following data is send to gRPC client, after deserialization at server side, we are losing choice field info. And as the field is marked as “required” field in model, any such request forwarded to controller / snappi would be validated and marked as error. This is causing gRPC client / server to fail, after addition of recent validation in Athena.

"flows": [
                {
                                "name": "p1->p2",
                                "tx_rx": {
                                                "choice": "port",
                                                "port": {
                                                                "tx_name": "p1",
                                                                "rx_name": "p2"
                                                }
                                }
                }

Now we can do following, to solve the issue:

Start user configurable enum value from 1. Drawback, we need to introduce a fake enum, as enum value of 0 is must.

message FlowTxRx {
  Choice.Enum choice = 1;
  FlowPort port = 2;
  FlowDevice device = 3;
 
  message Choice { enum Enum {
    reserved_enum = 0;
    port = 1;
    device = 2;
  } }
}

Make the field value using enum type as optional. But the field is marked required in the model. May be it can be marked as optional and assign a default value for the field. This has to be done on any field using zero indexed enum.

message FlowTxRx {
  optional Choice.Enum choice = 1;
  FlowPort port = 2;
  FlowDevice device = 3;
 
  message Choice { enum Enum {
    port = 0;
    device = 1;
  } }
}

We have also issue with fields which are not “required” in the model. If user sets some value equivalent to zero, those setting will be lost during deserialization. I guess, all such non-required fields will be marked as “optional” in the proto definition.

This is the general pattern used in go to allow optional arguments. We should use this pattern and remove hard coded parameters in generated code.

type Opts struct {
	httpServer         string
	debug              bool
}

func NewClient(opts *Opts) (*ApiClient, error) {
        if opts == nil {
                // defaults
                opts = &Opts{httpServer: "https://localhost", debug: true}
        }
        // rest of the code
}

func main() {
        // defaults
        api, err := NewClient()
        // non-defaults
        api, err := NewClient(&Opts{httpServer: "https://192.168.0.1", debug: false})
}

What: choice is currently being added as a property to every object when it is not a valid member. This is causing serialization/deserialization errors in some models.

Is your feature request related to a use case? Please describe.
When constructing and/or getting an object all fields should be pre-populated with defaults.

Describe the solution you'd like
The go ux sdk generator should generate code in the New... or getter functions that populates any leaf fields that have default values using the go ux sdk setter functions.

// Sample use case demonstrating the same library being used in a client and server.
// This removes the need of populating defaults during serdes

// client creates a minimal config
config = api.NewPrefixConfig()
warnings, err := api.SetConfig(config)

// server about what has been serialized as node access
// autmatically creates defaults that have not been supplied
http.HandleFunc("/config", func(w http.ResponseWriter, r *http.Request) {
    switch r.Method {
    case http.MethodPost:
        body, _ := ioutil.ReadAll(r.Body)
        config := api.NewPrefixConfig()
        config.FromJson(string(body))
        // validate what has been deserialized
        config.Validate()
        // use defaults if content is missing
        assert.Equal(config.A(), "asdf")
        w.Header().Set("Content-Type", "application/json")
        response := httpServer.Api.NewSetConfigResponse()
        switch httpServer.Config.Response() {
        case PrefixConfigResponse.STATUS_200:
            response.SetStatusCode200([]byte("Successful set config operation"))
            w.WriteHeader(http.StatusOK)
            w.Write([]byte(response.StatusCode200()))

Sample generated go code

func (api* openapiartApi) NewPrefixConfig() PrefixConfig {
	newObj := &prefixConfig{obj: &sanity.PrefixConfig{}}
	newObj.SetA("asdf")
	newObj.SetB(65.0)
        // ... remainder of required properties here
	return newObj
}

Describe alternatives you've considered
Populating the wrapped protobuf message will require additional utility code overhead while the use of go ux sdk setters does not.

f-strings are support only from python3.6+ and not supported in python2.7.
failing the build in snappi.

Current snappi generator incorporate this within init.py which is missing in openapiart.

from .snappi import api

I think it will better to add that to directly access api method. Otherwise we need to extract package to get api.

Maybe something like this:

import snappi
api = snappi.api()

api.info()
# output
{
    "title": "Open Traffic Generator API",
    "version": "0.4.5",
    "url": "https://github.com/open-traffic-generator/models",
    "license": "https://opensource.org/licenses/MIT"
}

When openapiart is installed via pip and as the package reside under site-packages. fetching dir name with file causing to fetch the site-package path of openapiart and all the go sdk stuff is getting created under it.

OpenApiArt needs to provide a go gettable package for sanity/example purposes.
Running the OpenApiArt python module within the openapiart repository should create an OpenApiArt go ux sdk using the internal models.

A user should be able to get the package and execute the unit tests.

go get "github.com/open-traffic-generator/openapiart/pkg"
cd $GOPATH/pkg/mod/github.com/open-traffic-generator/openapiart/pkg
go test -v

Using openapiart go pkg generation is achieved by specifying the following parameters:

openapiart.OpenApiArt(
    protobuf_package_name="sanity",
    go_sdk_package_dir="github.com/open-traffic-generator/openapiart/pkg",
    go_sdk_package_name="openapiart"
)

Proto file should include the following:

option go_package = "<go_sdk_package_dir>";

The OpenApiArt go sdk generator should create the following directory structure and contents based on the input parameters above:

/<go_sdk_package_dir>
    <go_sdk_package_name>.go 
    /<protobuf_package_name>
        <protobuf_package_name>_pb.go
        <protobuf_package_name>_grpc.pb.go

Tests should be created under the <go_sdk_out> directory and use the following import notation to ensure black box testing.
This allows tests to be run after generation in a dev/non-dev environment.

import . "github.com/open-traffic-generator/openapiart/<go_sdk_output_dir>"

error / warning are not included in following structs. Same issue for other response code as well.
message SetConfigResponse {
message StatusCode200 {
}
message StatusCode400 {
}
message StatusCode500 {
}
oneof statuscode {
StatusCode200 statuscode200 = 1;
StatusCode400 statuscode400 = 2;
StatusCode500 statuscode500 = 3;
}
}

A Property with default is being set even though it has required flag, I think we can limit this with two options

1. update model guide to either include default or required.
2. Throw an exception on the client if a property has both required and default.

Ex: https://github.com/open-traffic-generator/openapiart/blob/main/openapiart/tests/config/config.yaml#L91

    EObject:
      x-include:
      - '../common/common.yaml#/components/schemas/GlobalObject'
      required: [e_a, e_b]
      properties:
        e_a:
          type: number
          format: float
          default: 1.0
        e_b:
          type: number
          format: double
          default: 2.0```

We should not use the extension, and rather rely on following simple algorithm:

• dump current version (from project dir)
• check if a tag exists for given version
• create a new tag, release and publish to pypi if tag doesn't exist

@ajbalogh please review the steps so we can incorporate this across all repositories.
Check example .yml in https://github.com/open-traffic-generator/snappi-convergence/blob/master/.github/workflows/cicd.yml#L22

Slice types are not updated for PrefixConfig in generated go sdk

models:

        d_values:
          description: A list of enum values
          type: array
          items:
            type: string
            enum: [a, b, c]

Slice d_values obj is not added for PrefixConfig in generation

func (obj *prefixConfig) C() int32 {
	return obj.obj.C
}

func (obj *prefixConfig) SetC(value int32) PrefixConfig {
	obj.obj.C = value
	return obj
}

func (obj *prefixConfig) E() EObject {
	if obj.obj.E == nil {
		obj.obj.E = &sanity.EObject{}
	}
	return &eObject{obj: obj.obj.E}

}

common.go file has go common code which will be added to the generated go package. *.go files needs to be made available when openapiart is installed via pypi.

Issue(s):

• Type checking has been implemented in the open-traffic-generator/snappi repository and needs to be merged into the openapiart repository.

import openapiart

openapiart.OpenApiArt(
api_files=['C:/Users/rangabar.KEYSIGHT/Documents/GitHub/models/api/api.yaml'], #--> snappi models
python_module_name='throttle',
output_dir='./artifacts',
protobuf_file_name='throttle'
)

################### info.yaml #################
openapi: 3.1.3

info:
title: Open Traffic Generator API
description: |-
The Open Traffic Generator API consists of the following features:
- a common Traffic Generator API
- an Advanced Metrics API feature (under review)

Contributions can be made in the following ways:
- [open an issue](https://github.com/open-traffic-generator/models/issues) in the models repository
- [fork the models repository](https://github.com/open-traffic-generator/models) and submit a PR

version: 0.4.0
contact:
url: https://github.com/open-traffic-generator/models
license:
name: MIT
url: https://opensource.org/licenses/MIT
x-model-guide: https://github.com/open-traffic-generator/models/blob/master/MODEL-GUIDE.md

servers:

• url: /

paths: {}

####################################

------------------------------------error output-----------------------------------
grpc_tools.protoc args: C:\Users\rangabar.KEYSIGHT\Desktop\snappi_env\openapiart\Scripts\python.exe -m grpc_tools.protoc --python_out=C:\Users\rangabar.KEYSIGHT\Desktop\openapiart\artifacts\throttle --grpc_python_out=C:\Users\rangabar.KEYSIGHT\Desktop\openapiart\artifacts\throttle --proto_path=C:\Users\rangabar.KEYSIGHT\Desktop\openapiart\artifacts throttle.proto
throttle.proto:4464:175: Expected ";".
throttle.proto:12971:26: Missing field number.
throttle.proto:12986:18: Missing field number.
throttle.proto:13001:24: Missing field number.
throttle.proto:13016:20: Missing field number.
throttle.proto:13031:26: Missing field number.
throttle.proto:13036:29: Missing field number.
throttle.proto:13048:24: Missing field number.
throttle.proto:13059:26: Missing field number.

openapiart generation throws protobuf parsing error.

Possible functionality as follows:

OpenApiArt(api_files=["./api/api.yaml"], protobuf_name="sanity", artifact_dir="./art") \
    .GeneratePythonSdk(python_module_name="sanity") \
    .GenerateGoSdk(package_dir="github.com/open-traffic-generator/openapiart/pkg", package_name="openapiart")

There is a behavior change in append method on Iter object and breaking the existing behavior in snappi.
currently commented the append method portion in generator.py and shall revisit on the same.

It looks like snappi==0.4.2 and onward breaks when serializing configurations with incrementing MAC addresses.

grpcio and grpcio-tools are pretty heavy (and probably rely on c extensions) - anybody who tries to install generated package will have to install these as dependencies

Currently ENUMs look like this:

message LagProtocol {
  option (msg_meta).description = "Description missing in models";
  enum Choice {
    CHOICE_UNSPECIFIED = 0;
    CHOICE_LACP = 1;
    CHOICE_STATIC = 2;
  }
  optional Choice choice = 1 [
    (fld_meta).default = "Choice.CHOICE_LACP",
    (fld_meta).description = "The type of LAG protocol."
  ];
  optional LagLacp lacp = 2 [
    (fld_meta).description = "Description missing in models"
  ];
  optional LagStatic static = 3 [
    (fld_meta).description = "Description missing in models"
  ];
}

when serializing to JSON, following happens:

• enums are all upper case
• enums are prefixed with parent's name

This makes the JSON incompatible when deserializing to snappi object.

We have plan for >= 3.8 then fine. Otherwise we need to put some workaround

try:
    from typing import Literal
except ImportError:
    from typing_extensions import Literal

@ajbalogh FYI