ciscodevnet / ydk-gen Goto Github PK

Currently, there is very little documentation for the DELETE object. The documentation needs to specify when and how to use the DELETE object. Also, whether to use it in a CRUD update operation or elsewhere. Documentation can be similar to read filter

Candidates include

1. gRpc
2. Restconf

According to this blog post, when submitting pull request, the variable TRAVIS_BRANCH is not reflecting the current branch, which will fails the CI for pull request, even though the CI for push pass.

This can be easily reproduced. While decoding the namespace lookup for nodes in the submodule fails because the submodule is not included in the generated namespace list

From Xiaoqin:

I read from this style guide: https://google.github.io/styleguide/pyguide.html#Strings
Avoid using the + and += operators to accumulate a string within a loop. Since strings are immutable, this creates unnecessary temporary objects and results in quadratic rather than linear running time. Instead, add each substring to a list and ''.join the list after the loop terminates (or, write each substring to a io.BytesIO buffer).

Yes: items = ['<table>']
     for last_name, first_name in employee_list:
         items.append('<tr><td>%s, %s</td></tr>' % (last_name, first_name))
     items.append('</table>')
     employee_table = ''.join(items)

No: employee_table = '<table>'
    for last_name, first_name in employee_list:
        employee_table += '<tr><td>%s, %s</td></tr>' % (last_name, first_name)
    employee_table += '</table>'

By changing += and + operators to list join, it might increase the speed for documentation.

As suggested by Santiago:

• Pip will automatically take care of the dependencies
• Also, we can potentially create man pages with documentation
• This way, we don't need install.sh to install the dependencies/create virtual env

If someone already has YDK installed and want to upgrade to a newer version of YDK, need to come up with some workflow in the README or some other document. Once this is available on 'pip install', may be this can happen little more smoothly

Consider the following model

container A {
    container B {
      presence "Active";

     list C {

     }
} 

When creating the above entity tree using the api

a = A()

The reference to B is set to None. In other words references to presence classes are set to None and have to explicitly initialized by the user. This is necessary in Create/Update

However when used as a filter in read we need to make sure that this read ?

Background

Currently, YDK covers all the YANG model APIs, including openconfig, IETF, Cisco etc under the same python package and this is published as the ydk-py python package. The main reason why bundles are needed is that there needs to be a way to separately generate python packages for YDK model APIs for various YANG module repositories like openconfig, IETF, Cisco XR, Cisco XE etc. This will have more real world uses.

Bundling is meant as a extension of and replacement to the profile mechanism, which ydk-gen currently makes use of. Currently, ydk-gen generates a set of model APIs based on a profile and combined with the runtime YDK components (providers & services), creates a pip installable package (model APIs + YDK providers & services) called ydk-py.

Proposal

With the introduction of bundles, ydk-gen will no longer generate a pip package called ydk-py. Instead the new pip package will only contain the model API's for a given set of YANG models which are specified in the bundle JSON file. For example, ydk-gen can generate a pip installable package (model APIs) called ietf-bundle.

Currently, ydk-gen operates in the below way:

1. Parse profile JSON
2. Download yang models
3. Using pyang, parse YANG files and return modules which are trees of Statement python objects
4. Generate expanded YDK API model Packages from the above Statement trees
5. Pass the ydk Packages to the PythonModulePrinter to print the YDK model API files

Package structure looks like below:

ydk-py
 |
 |-ydk
    |-models
       |
       |- ietf_interface
       | - ietf_system
    |-providers
    |-services
 |-samples

By introducing the concept of bundling, the flow will be like the below:

Resolve bundle (This sub-task is tracked in #26)

1. Parse bundle JSON
2. Download yang models (e.g. openconfig yang models) specified in the bundle plus any dependent YANG models from a different bundle (e.g. IETF yang models) which can also be specified in the main bundle

Build API model for bundle(s)
3. Using pyang, parse YANG files and return modules which are trees of Statement python objects
4. Generate expanded YDK API model Packages from the above Statement trees. Here, each bundle may utilize different code generation strategies, depending on the YANG model structure of the models in the bundle. For example, bundles can make use of choice statement inheritance or collapse containers with single list
5. Assign the list of YDK Packages as children of Bundle objects (by making use of the bundle JSON file)

Print model APIs for bundle(s)
6. Pass the ydk Bundles to the YDK Printer to print the YDK model API files for each bundle and generate the pip installable bundle (can be a python package and in the future, can be C++ library/ Ruby gem)

Package structure will look like below:

ietf-bundle
 |
 |-models
    |
    |- ietf_interface
    | - ietf_system
 |-samples

This enables users to handle below use cases as part of YDK app maker (YAM)

• Have netconf, restconf payload and want to migrate to YDK based app with minimal effort via tool.
• Looking for an educational channel on how to use YDK objects.
• Yang data model payload transcoding (convert XML to YDK to JSON to YDK python to .... or vice versa)

class CodeGenerationServiceProvider(object):
    """ Code generation provider"""
   def __init__(self, type):
        '''type is 'xml' or 'json' '''

class CodeGenerationService(object):
    """ Code generation service to generate code from XML/JSON payload """
   def __init__(self):
        pass
    def generate(self, provider, payload):
        '''This will return a python script snippet which can be used in a YDK app'''

It is preferable to continue using the existing service/provider abstraction

Blocked by #46

Currently CRUD operations work on a single entity. This does not allow a single create operation to work on multiple entities that do not have a common parent. By introducing a Top entity which represents that datastore (for lack of better term), we can allow these operations to happen

Proposal is to create a class per module (for those modules/packages that have classes/enums defined in them).
These module classes will have properties for the top level elements for example the oc-bgp module will have a top level class Module this class will have one property which is hte top level Bgp class that corresponds to the container bgp.

Note this class must be at the same level as the Bgp classes and should appear after the top level classes have been defined. Further more this class should not have any nested type definitions

Example 👎

module A {

container A {
..
}
leaf b;

list C {
...
}
}

will produce

class A(object): --> container A

class C(object): --> list C

Now the new module class

class AModule(object):
def init(self):
self.a = A()
self.b = None --> for leaf B
self.c = YList() ---> list of C's

When retriving data from a model with a lot of oper data, ncclient will timeout causing TimeoutExpiredError to be thrown.

Establishing connection with device 10.8.19.1:830 using ssh:
Connection established
Traceback (most recent call last):
File "yang_mplste_oper.py", line 53, in
main()
File "yang_mplste_oper.py", line 44, in main
crud_service.read(session, mpls_te)
File "/home/gregorbr/yang_test/local/lib/python2.7/site-packages/ydk/services/crud_service.py", line 127, in read
payload = self.execute_crud_operation_on_provider(provider, read_filter, 'READ', only_config)
File "/home/gregorbr/yang_test/local/lib/python2.7/site-packages/ydk/services/crud_service.py", line 140, in execute_crud_operation_on_provider
only_config
File "/home/gregorbr/yang_test/local/lib/python2.7/site-packages/ydk/services/service.py", line 31, in execute_payload
payload)
File "/home/gregorbr/yang_test/local/lib/python2.7/site-packages/ydk/providers/_provider_plugin.py", line 330, in execute_operation
reply_str = sp_rpc._request(payload)
File "/home/gregorbr/yang_test/local/lib/python2.7/site-packages/ncclient/operations/rpc.py", line 342, in _request
raise TimeoutExpiredError('ncclient timed out while waiting for an rpc reply.')
ncclient.operations.errors.TimeoutExpiredError: ncclient timed out while waiting for an rpc reply.

Proposed fix:

session = NetconfServiceProvider(timeout=300,....) #add new argument to set timeout

I ran across an issue where the yang models were incorrectly loaded/wrong version of yang models were loaded to the server. So, when I ran the sanity tests, the validation mechanism did not catch the error since the payload was valid. However, the server rejected them because of mismatched model. For this case, I think the error-tag and error-info from the RPC error can be captured in the exception raised, so that the user has a clear idea why the server rejected the request.

As part of this, need some samples for read filter using netconf service.

When trying to use YDK on ietf-interfaces.yang, there were issues encountered with identities defined in external modules (iana-if-type.yang). The prefix being used in the encoded RPC was wrong. Also, issues were seen with encoded enums in ietf-system.yang.

Possible candidates

• C++
• Java

Currently, the docker CI script uses pre-generated fxs files. However, the yang files should be used at build time to build fxs files. This will make sure if any changes are made to the test yang files, the changes are pulled in for each travis build

This task will generate a unit test that creates an object graph that is part of a payload of an RPC, serializes the RPC then deserailizes the RPC and extracts the object graph. The object graphs are then compared to ensure that they are the same.

Note this tests

1. The encoder/decoder logic
2. Exercises all the generated api making sure that it confirms to python syntax

~/ydk-gen > python ./generate.py -p --no-doc --profile profiles/ydk/ydk_0_4_0.json --verbose --groupings-as-class

Traceback (most recent call last):
File "./generate.py", line 151, in
options.groupings_as_class)
File "/ydkgen/init.py", line 316, in generate
modules)
File "/ydkgen/api_model.py", line 1279, in generate_grouping_class_api_model
create_grouping_class_api_model(m, p)
File "/ydkgen/api_model.py", line 1189, in create_grouping_class_api_model
stmt.i_groupings[grouping_name], element)
File "/ydkgen/api_model.py", line 1260, in create_grouping_class_api_model
create_grouping_class_api_model(child_stmt, element)
File "/ydkgen/api_model.py", line 1260, in create_grouping_class_api_model
create_grouping_class_api_model(child_stmt, element)
File "/ydkgen/api_model.py", line 1223, in create_grouping_class_api_model
enum_type = _get_enum_type_stmt(stmt)
File "/ydkgen/api_model.py", line 766, in
_get_enum_type_stmt = lambda stmt: _get_type_stmt(stmt, EnumerationTypeSpec)
File "/ydkgen/api_model.py", line 753, in _get_type_stmt
type_stmt = type_spec.i_target_node.search_one('type')
AttributeError: 'PathTypeSpec' object has no attribute 'i_target_node'

When running a series of YDK-Py apps, you may get the following thread exception:

Exception in thread session (most likely raised during interpreter shutdown):
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
File "/usr/local/lib/python2.7/dist-packages/ncclient/transport/ssh.py", line 554, in run <type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'debug'

May be related to ncclient:

Exception in thread session (most likely raised during interpreter shutdown):
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
File "/home/gitlab-runner/builds/77b7020d/0/ydk-dev/ydk-gen/myenv/local/lib/python2.7/site-packages/ncclient/transport/ssh.py", line 556, in run
File "/home/gitlab-runner/builds/77b7020d/0/ydk-dev/ydk-gen/myenv/local/lib/python2.7/site-packages/ncclient/transport/ssh.py", line 301, in close
File "/home/gitlab-runner/builds/77b7020d/0/ydk-dev/ydk-gen/myenv/local/lib/python2.7/site-packages/paramiko/transport.py", line 644, in close
File "/home/gitlab-runner/builds/77b7020d/0/ydk-dev/ydk-gen/myenv/local/lib/python2.7/site-packages/paramiko/transport.py", line 1536, in stop_thread
File "/usr/lib/python2.7/threading.py", line 960, in join
File "/usr/lib/python2.7/threading.py", line 363, in wait
<type 'exceptions.ValueError'>: list.remove(x): x not in list

Currently, there are a lot of steps which need to be manually carried out to install ydk-gen or ydk-py. There needs to be a simple bash script which takes care of everything (including installing dependencies and setting up environment). This will make it easy for users to install and use ydk-gen and/or ydk-py

Describe how a developer would handle incompatible changes in a model.

Move presence information to meta data, add external is_presence method

In the documentation for the corresponding python class, need to specify which of the leafs for a yang list are keys.

For example, for the XR native ISIS model, in the class Isis > Instances > Instance, the instance_name should be identified as the key

We have left boilerplate code in the setup.py that we generate for a ydk-py that has:

        # Specify the Python versions you support here. In particular, ensure
        # that you indicate whether you support Python 2, Python 3 or both.
        #'Programming Language :: Python :: 2',
        #'Programming Language :: Python :: 2.6',
        'Programming Language :: Python :: 2.7',
        'Programming Language :: Python :: 3',
        'Programming Language :: Python :: 3.2',
        'Programming Language :: Python :: 3.3',
        'Programming Language :: Python :: 3.4',

The version requirement we have for ncclient is not a py3-compatible version, and I don;t think we have ever tested with py3, so I think we should take out the 3, 3.2, 3.3 and 3.4.

Background

Currently the DELETE object supports deletion on leaf and container level for CRUD update operation, which provide user with flexibility to delete and create attribute at the same time. However we lack the support for using DELETE object on list.

Proposal

We could add an attribute or property _delete on python class created from list statement, and with this attribute being set as DELETE object, we could provide user the capability to delete list element with specified list key. For example,

    bgp_cfg = bgp.Bgp()

    bgp_cfg.global_.config.as_ = 65001

    ipv4_afsf = bgp_cfg.global_.afi_safis.AfiSafi()
    ipv4_afsf.afi_safi_name = 'ipv4-unicast'
    ipv4_afsf.config.afi_safi_name = 'ipv4-unicast'
    ipv4_afsf.config.enabled = True

    ipv6_afsf = bgp_cfg.global_.afi_safis.AfiSafi()
    ipv6_afsf.afi_safi_name = 'ipv6-unicast'
    ipv6_afsf.config.afi_safi_name = 'ipv6-unicast'
    ipv6_afsf.config.enabled = True
    ipv6_afsf.delete = DELETE()

    bgp_cfg.global_.afi_safis.afi_safi.append(ipv4_afsf)
    bgp_cfg.global_.afi_safis.afi_safi.append(ipv6_afsf)

CRUD update operation on bgp_cfg will delete this ipv6 afsf and create ipv4 afsf.

Need to look into these issues:

• the examples are great and for the run-of-the-mill code, the ramping up is good
• For something more fancy, I.e., identity refs, things are more intricate and not very intuitive in the example I looked at
• Debugging can be challenging. E.g., the encoder was complaining that “name” could not be encoded. (I had three “name” at different levels of the hierarchy, so I had to dig which of them was the reported one)
  • To debug, I had to modify the generated code to get extra info
  • The leaf is a yang string. The value I had was unicode in python (I had read it from a file in json) and the encoder was checking for str
  • Exceptions like 'invalid hierarchy' need a bit more detailed description on how to fix these
• Constructors that take values for leaves may be useful

From Lily's suggestion:

Every python class is getting a class docstring written into it. I was wondering if this is still necessary since the exact same docstring is already getting written into each file’s respective documentation file?

Currently, git push to the github repository gives the below warning:

remote: warning: GH001: Large files detected. You may want to try Git Large File Storage - https://git-lfs.github.com.
remote: warning: See http://git.io/iEPt8g for more information.
remote: warning: File ydk/models/infra/Cisco_IOS_XR_infra_policymgr_oper.py is 78.48 MB; this is larger than GitHub's recommended maximum file size of 50.00 MB
remote: warning: File ydk/models/ipv4/Cisco_IOS_XR_ipv4_bgp_oper.py is 74.33 MB; this is larger than GitHub's recommended maximum file size of 50.00 MB

Need to come up with a solution to fix these warnings

I noticed that these dependencies are in the requirements.txt for ydk-py:

Sphinx==1.4a1
sphinx-rtd-theme==0.1.9
XlsxWriter==0.7.3
GitPython==1.0.1

These seem like ydk-gen dependencies that should be removed?

Currently the url for github repo used in .travis.yml is hardcoded, so we need to change the pointer to master repo after each merge, and before each commit for forked repo, which is inconvenient. There might be a better way to handle it.

Make it obvious in documentation. Indicate which elements are mutually exclusive

Also, if the --choice-subsstatements-as-subclasses option is chosen when running ydk-gen, all the cases of a choice statement should be generated as subclasses of a parent class.

Currently, the cases of a choice statement are modeled as Properties. So, below yang produces below python code.

choice rule-type {
          description
            "This choice matches if all leafs present in the rule
             match the request.  If no leafs are present, the
             choice matches all requests.";
          case protocol-operation {
            leaf rpc-name {
              type union {
                type matchall-string-type;
                type string;
              }
              description
                "This leaf matches if it has the value '*' or if
                 its value equals the requested protocol operation
                 name.";
            }
          }
          case notification {
            leaf notification-name {
              type union {
                type matchall-string-type;
                type string;
              }
              description
                "This leaf matches if it has the value '*' or if its
                 value equals the requested notification name.";
            }
          }
          case data-node {
            leaf path {
              type node-instance-identifier;
              mandatory true;
              description
                "Data Node Instance Identifier associated with the
                 data node controlled by this rule.

                 Configuration data or state data instance
                 identifiers start with a top-level data node.  A
                 complete instance identifier is required for this
                 type of path value.

                 The special value '/' refers to all possible
                 datastore contents.";
            }
          }
        }

self.notification_name = None
self.path = None
self.rpc_name = None

If this option is chosen, below python code will be produced instead :

self.rule_type = RuleType()

class RuleType(object):
     pass

class NotificationName(RuleType):
    pass

class Path(RuleType):
    pass

class(RuleType):
    pass

RuleType is not from the choice, rather it should be from the container or list that is the parent of the choice. No class should be generated for the choice. (In the case where choice is a top level element under the module, the case classes would inherit themodule class)

Note in the case there are 2 choices under the given statement , this generation algorithm should not be used

For instance, documentation for enums in Cisco_IOS_XR_ip_ntp_cfg is generated correctly. However, documentation for enums in Cisco_IOS_XR_clns_isis_cfg module points to a non-existent files.

Text and samples seem incomplete: http://ydk.cisco.com/py/docs/types_doc.html We should either expand or remove.

The version of the generated SDK shouldn't be hard coded in the SDK files. The generation profile already specifies the version number. That version should be added to the SDK package during generation.

• profile file should have the version number for the ydk-py which is being generated
• ydk-py should document somewhere the link to the ydk-gen commit which was used to generate it

Seeing the following error:

File "/home/gregorbr/yang_test/local/lib/python2.7/site-packages/ydk/services/crud_service.py", line 129, in read
return self.create_read_return_value(payload, read_filter)
File "/home/gregorbr/yang_test/local/lib/python2.7/site-packages/ydk/services/crud_service.py", line 162, in create_read_return_value
DmTree._bind_to_object(payload, entity)
File "/home/gregorbr/yang_test/local/lib/python2.7/site-packages/ydk/_core/_dmtree.py", line 157, in _bind_to_object
DmTree._bind_to_object_helper(curr_rt, entity, pretty_p='|-')
File "/home/gregorbr/yang_test/local/lib/python2.7/site-packages/ydk/_core/_dmtree.py", line 197, in _bind_to_object_helper
DmTree._bind_to_object_helper(rtchild, child, pretty_p+'-l')
File "/home/gregorbr/yang_test/local/lib/python2.7/site-packages/ydk/_core/_dmtree.py", line 185, in _bind_to_object_helper
DmTree._bind_to_object_helper(rt[0], instance, pretty_p+'-|')
File "/home/gregorbr/yang_test/local/lib/python2.7/site-packages/ydk/_core/_dmtree.py", line 185, in _bind_to_object_helper
DmTree._bind_to_object_helper(rt[0], instance, pretty_p+'-|')
File "/home/gregorbr/yang_test/local/lib/python2.7/site-packages/ydk/_core/_dmtree.py", line 268, in _bind_to_object_helper
entity.dict[member.presentation_name] = DmTree._to_real_union_type_helper(rt, member,entity)
File "/home/gregorbr/yang_test/local/lib/python2.7/site-packages/ydk/_core/_dmtree.py", line 93, in _to_real_union_type_helper
elif contained_member.ptype == 'int' and rt[0].text is not None and rt[0].isidigit():
AttributeError: 'lxml.etree._Element' object has no attribute ‘isidigit'

Proposed fix:

rt[0].isdigit() should be rt[0].text.isdigit

Provide appropriate infrastructure classes to parse typical telemetry data being exported in protobuf format into ydk-py objects.

When trying to connect via tcp, ydk throws an error:

Establishing connection with device 10.8.19.3:830 using tcp:
Traceback (most recent call last):
File "yang_mplste_oper.py", line 53, in
main()
File "yang_mplste_oper.py", line 39, in main
session = NetconfServiceProvider(address=o.host, port=o.port, username = o.username, password = o.password, protocol = o.proto)
File "/home/gregorbr/yang_test/local/lib/python2.7/site-packages/ydk/providers/netconf_provider.py", line 66, in init
self._session_config,
AttributeError: 'NetconfServiceProvider' object has no attribute ‘_session_config'

Looks like NetconfServiceProvider passes tcp to ncclient in which it is not supported.

Proposed fixed:

Disable tcp support.

Background

Currently, there is a problem with the size of the individual python modules generated for some of the YANG models. For example, currently, ned.py is around 125 MB. This make it difficult for someone to read the code and makes it hard to even load the file in some editors like eclipse.

Proposal

We can make use of the fact that python allows a single module to be split into multiple source files. Then a __init__.py file can be used to unify the different source files into a single logical module. See this link to python cookbook for more details.

So, for example, this is the current organization of the ned module:

~/ydk-gen/gen-api/python/ydk/models > ls -l ned
total 177288
-rw-r--r--  1 abhirame  staff         0 Mar  4 11:49 __init__.py
drwxr-xr-x  4 abhirame  staff       136 Mar  4 11:50 _meta/
-rw-r--r--  1 abhirame  staff  90770431 Mar  4 11:50 ned.py
~/ydk-gen/gen-api/python/ydk/models > cat ned/__init__.py 
~/ydk-gen/gen-api/python/ydk/models >

Using the organization from the python cookbook, the below could be the new organization:

~/ydk-gen/gen-api/python/ydk/models > ls -l ned
total 177288
-rw-r--r--  1 abhirame  staff         0 Mar  4 11:49 __init__.py
drwxr-xr-x  4 abhirame  staff       136 Mar  4 11:50 _meta/
-rw-r--r--  1 abhirame  staff  70431 Mar  4 11:50 ned_native_service.py
-rw-r--r--  1 abhirame  staff  70431 Mar  4 11:50 ned_native_archive.py
-rw-r--r--  1 abhirame  staff  70431 Mar  4 11:50 ned_native_banner.py
...

~/ydk-gen/gen-api/python/ydk/models > cat ned/__init__.py 
from .ned_native_service import Native.Service
from .ned_native_archive import Native.Archive
from .ned_native_banner import Native.Banner
...
~/ydk-gen/gen-api/python/ydk/models >

This issue tracks reading bundle files written in json and resolving all the models and bundles that are referenced in these files as per the ydk model in the metaservice branch

In the presence of multiple classes, the top container seems to always be listed last which greatly impacts the usability of the documentation for models make heavy use of enums (e.g. Cisco_IOS_XR_clns_isis_cfg module).

We need to provide support for config replace capability (both global and per-model). The current CRUDService abstraction will not meet the operational requirements of customers. The requirement for config replacement is expected to increase in popularity and the lack of this functionality will hinder adoption by customers.

Per-model config replace: Given a list of config objects, replace the config of that specific list of models with the provided objects.

Global config replace: Given a list of config objects, replace the entire config of the device with the config data in those models. Any other models/objects not provided as input will be deleted.

Consider the following pattern in yang

              list bundle {
                .....
                container modules {
                        list module {
                        ...

In the expanded API generations strategy the source file will be encoded as follows

              class Bundle(object): --> corresponds to the list bundle
                def __init__(self):
                    self.modules = Modules() 

                class Modules(object): --> corresponds to the container modules
                    def __init__(self):
                        self.module = YList() --> corresponds to the list Module

                    class Module(object): --> corresponds to the list module     

If this leaf is set the generated api will skip the container modules and have the class Module as a child of the Bundle class like this

                   class Bundle(object):  ---> corresponds to the list bundle
                      def __init__(self):
                         self.modules = YList() --> corresponds to the list Module

                      class Module(obect):
                           ...

Background

Currently, the providers and services API's are organized into separate modules. So, anyone writing an app has to write multiple import statements

from ydk.services import CRUDService
from ydk.providers import NetconfServiceProvider

In the future if we add more services and providers, the app would have to import the new classes from separate modules as well.

Proposal

We can make use of the fact that python allows a single module to be split into multiple source files. Then a __init__.py file can be used to unify the different source files into a single logical module. See this link to python cookbook for more details.

By adding an __init__.py to the ydk directory and loading all the services and providers module in it, we can unify the whole of ydk into a single logical module.

ydk/__init__.py will look something like the below:

from .providers import NetconfServiceProvider
from .services import CRUDService, ExecutorService

So, in an app, all the classes needed for that app can be imported with a single statement:

from ydk import CRUDService, NetconfServiceProvider, Empty