yangcatalog / yang-validator-extractor Goto Github PK

After a certain amount of API calls to the API endpoint /api/rfc/< rfc > - which is used in health check functionality, YANG validator starts to respond with following error:
RecursionError: maximum recursion depth exceeded while calling a Python object

As in title

Hi again,

Aside from the previous issue, we are trying to investigate a validation error, and we are not fully understanding what is happening.

When validating draft-ietf-ccamp-flexigrid-tunnel-yang, we have the following yanglint error:

libyang err : Invalid default - value does not fit the type (Invalid identityref "flexi-ch-spc-6p25ghz" value - identity not found in module "ietf-flexi-grid-tunnel".). (Schema location /ietf-te:te/tunnels/tunnel/secondary-reverse-paths/secondary-reverse-path/path-out-segment/label-restrictions/label-restriction/label-step/technology/ietf-flexi-grid-tunnel:flexi-grid/flexi-grid-channel-spacing.)

But in the IETF tracker, also validated with yanglint (see the validation output), there is no error, and we are not really sure why validation outputs are different (and which we should trust :D).

Thank you in advance for your help!

as in title

Draft names are a well-known set of the file names that can be indexed into the ES index.
This can be used in YANG Validator UI for autocomplete functionality while validating draft by its name.
This task consists of following:

• UI changes: display suggestions fetched from backend
• backend: provide endpoint to get suggestions
• elasticsearch: create new ES index and index draft names
• crontab: create new script which will regularly check whether there are some new drafts to be indexed

as in title

An open API must be created / offered / documented / monitored to submit YANG modules, RFC/IETF drafts to the same tools as via the web interface to YANG validator. The returned information should also have the same content (presented as JSON).

-éric

Now that the yangvalidator is under yangcatalog, add a "yangvalidator" entry under https://github.com/YangCatalog
It should point to https://github.com/YangCatalog/bottle-yang-extractor-validator/issues (this comes from https://yangvalidator.com/yangvalidator/about)

Hi,
how the yangvalidator is supposed to handle imports? I guess that yangvalidator have some internal storage where validators search for imported modules, but what if user submit also one of the imported modules? Should the submitted schema be prefered to the schema in the storage? What if the version in storage is newer and import is without revision? And what if the revision of the submitted schema is not specified in the filename?

Specific example:

• storage contains [email protected]
• user submit files ietf-routing.yang and ietf-ipv4-unicast-routing.yang, revisons are not specified but revision of ietf-routing.yang is newer than 2016-11-03 and ietf-ipv4-unicast-routing imports ietf-routing without revision (so the newer)
• when the tools validate ietf-ipv4-unicast-routing and resolving ietf-routing import, are they supposed to
  • get revision just from the filename and suppose that if one ietf-routing has revision in filename, other ietf-routing without revision in filename simply does not have any revision (so user is supposed to submit correctly named files)?
  • parse all the ietf-routing files to get know the newest revision?
  • parse all accessible files (because ietf-routing.yang can be also named e.g. mySuperSchema.txt) to get the newest ietf-routing revision?

Display the current pyang, yanglint, ... versions on the input/home yangvalidator page.

https://www.yangcatalog.org/yangvalidator/
I see pyang, confdc. yanglint. We need yumaworks as well.
On top of that, we need the same output as the compilation-result in the metadata.
That means: we need pyang --ietf

By the way I have bigger issues with YangValidator. When I load ietf-system-capabilities@2020-03-08 into it I get

RecursionError at /yangvalidator/validator
maximum recursion depth exceeded

Hi,

Happy New Year!

As the title says, we had some issues with the online validator. When validating draft-ietf-ccamp-flexigrid-tunnel-yang, the online validator frontend just hangs and I can see in the browser DevTools that a request to the backend returned code 500.

Doing a further investigation, I just noticed that, when submitting the YANG code without the revision-date (see https://github.com/ietf-ccamp-wg/draft-ietf-ccamp-flexigrid-tunnel-yang/blob/master/ietf-flexi-grid-tunnel.yang#L9), the code gets validated. So my guess is that something is broken in the processing of the revision dates.

If we go to this page there is a message: "No modules were extracted using xym from /var/yang/ietf/my-id-archive-mirror/draft-claise-ipfix-information-model-rfc5102bis-01.txt"

It is better if we remove absolute path from this error message, so it will be only: "No modules were extracted using xym from draft-claise-ipfix-information-model-rfc5102bis-01.txt"

NOTE: Maybe we can display this on the UI side as info instead of error? Similar to this info

Following error is displayed after running the command docker logs yc-yangvalidator:

rsyslogd: imklog: cannot open kernel log (/proc/kmsg): Operation not permitted.
rsyslogd: activation of module imklog failed [v8.32.0 try http://www.rsyslog.com/e/2145 ]

Remove sync.sh script from Docker image build. This script pull rfc/draft documents using rsync however it is used in SDO analysis and running daily.
This will speed up Docker image build.

Fetch, extract , and validate doesn't produce any output when Upload Internet Draft option used.
Appreciate your suggestions and help.

Regards,
Greg

Is it possible to upload a set of zipped yang modules using the rest API and get them validated ?
What is the first part of the URL
https://www.yangvalidator.com/yangvalidator/rest
or
https://www.yangvalidator.com/yangvalidator

Thanks Balazs

Done here #67 and here #68

YANGvalidator doesn't produce any output when I upload my working version of the YANG model in IETF draft. Appreciate your help.

Regards,
Greg

I uploaded a zipped set of yang files that depend on each other (import each other). However imports between the uploaded files were not resolved. I got a lot of error messages like:

err : Importing "_3gpp-common-measurements" module into "_3gpp-common-managed-function" failed.

Is it possible to solve this? How?

upgrade and use pyang api instead of command line arguments

It seems that no YANG modules can be extracted from draft-ietf-lpwan-schc-yang-data-model-02.

Previously fine models fail with the following:
Traceback (most recent call last):
pyang", line 444, in
run()
pyang", line 219, in run
repos = pyang.FileRepository(path, no_path_recurse=o.no_path_recurse)
init.py", line 408, in init
location = pip.locations.distutils_scheme('pyang')
AttributeError: 'module' object has no attribute 'locations'
Much appreciate your help.

Regards,
Greg

Some steps in Dockerfile can be moved, so build will use caching more.
This can result in faster image builds.

#81 is marked as fixed but the home page of YANG validator has a link about the REST API leading to YC.O but not to the swagger documentation :-(

Tested on Chrome/Macos

when I click on the file listed under "Extracted 1 YANG modules" I just get taken to the same page I'm already on...

Need to get the xym extraction errors in
1. tracker => xym error
2. yangcatalog.org/yangvalidator.and yangvalidator.org

ex:

These errors are available at temp_runIETFstats.txt

Yangvalidator is no longer based on the bottle framework, and the naming should be changed to reflect this.

ietf-ospf.yang
Pyang Validation

ietf-ospf.yang:992: error: type "bandwidth-ieee-float32" not found in module ietf-routing-types
ietf-ospf.yang:996: error: type "bandwidth-ieee-float32" not found in module ietf-routing-types
ietf-ospf.yang:1009: error: type "bandwidth-ieee-float32" not found in module ietf-routing-types
ietf-ospf.yang:1966: error: identity "crypto-algorithm" not found in module ietf-key-chain

These are clearly in the latest versions of the modules.

The online yangvalidator throws errors on augments. Taken separately the augment statement validates, but when both augment statements are included in the same file, the validator chokes on them.

I have created four test yang files. Two of them validate in the online validator and two do not.

The test-ok.yang and test-ok-2.yang files work.

The test-ok file has only one augment (for /nw:networks/nw:network/nw:network-types/tet:te-topology)
The test-ok-2.yang file has only one augment (for /nw:networks/nw:network/nw:node/nt:termination-point/tet:te)
Taken separately they augments work.

The test-1.yang file has both augments Taken together, the online yanglint spits out:

libyang err : Augment target node "/nw:networks/nw:network/nw:network-types/tet:te-topology" from module "test-1" was not found. (/test-1:{augment='/nw:networks/nw:network/nw:network-types/tet:te-topology'})
libyang err : Augment target node "/nw:networks/nw:network/nw:node/nt:termination-point/tet:te" from module "test-1" was not found. (/test-1:{augment='/nw:networks/nw:network/nw:node/nt:termination-point/tet:te'})
YANGLINT[E]: Processing schema module from /var/yang/tmp/yangvalidator/yangvalidator-v2-workdir-BYiAZxUm/test-1.yang failed.

When I swap the order of the augments, I get different errors – which is strange. (test-2.yang)

libyang err : Augment target node "/nw:networks/nw:network/nw:node/te/te-node-attributes/connectivity-matrices/connectivity-matrix" from module "ietf-te-topology" was not found. (/ietf-te-topology:{augment='/nw:networks/nw:network/nw:node/te/te-node-attributes/connectivity-matrices/connectivity-matrix'})
libyang err : Augment target node "/nw:networks/nw:network/nw:node/te/tunnel-termination-point/local-link-connectivities" from module "ietf-te-topology" was not found. (/ietf-te-topology:{augment='/nw:networks/nw:network/nw:node/te/tunnel-termination-point/local-link-connectivities'})
libyang err : Augment target node "/nw:networks/nw:network/nw:node" from module "ietf-te-topology" was not found. (/ietf-te-topology:{augment='/nw:networks/nw:network/nw:node'})
libyang err : Augment target node "/nw:networks/nw:network/nw:node/te/information-source-entry/connectivity-matrices/connectivity-matrix" from module "ietf-te-topology" was not found. (/ietf-te-topology:{augment='/nw:networks/nw:network/nw:node/te/information-source-entry/connectivity-matrices/connectivity-matrix'})
YANGLINT[E]: Processing schema module from /var/yang/tmp/yangvalidator/yangvalidator-v2-workdir-fXryozgz/test-2.yang failed.

I have also attached the the full module that I'm trying to validate ([email protected])

BTW... I have yanglint v1 and yanglint v2 running locally in linux and all the yang files I have shared validate (and I have instance data that works too)
augtest.zip
.

thanks

also use socket files rather then port

Transfer yangvalidator.[org/com] to yangcatalog.

RFC 6021 contains two YANG modules but only one is processed

as in title

Problem:
yangvalidator app is currently running as root, which results in permission error in removeUnused.py script while trying to delete old yangvalidator cache files.

Traceback (most recent call last):
  File "/backend/utility/removeUnused.py", line 219, in <module>
    raise e
  File "/backend/utility/removeUnused.py", line 126, in <module>
    shutil.rmtree(os.path.join(yang_validator_cache, dir))
  File "/usr/local/lib/python3.9/shutil.py", line 718, in rmtree
    _rmtree_safe_fd(fd, path, onerror)
  File "/usr/local/lib/python3.9/shutil.py", line 675, in _rmtree_safe_fd
    onerror(os.unlink, fullname, sys.exc_info())
  File "/usr/local/lib/python3.9/shutil.py", line 673, in _rmtree_safe_fd
    os.unlink(entry.name, dir_fd=topfd)
PermissionError: [Errno 13] Permission denied: 'nokia-conf.yang'

These are helper files which are copied into cache directories during validation.

Possible solution:
If application will be running under yang user/group, these files will be also created/copied under this user.

The web tool reports descriptors appearing before a type as an error.

For example in a leaf
leaf foo {
description "example";
type uint32;
}
one gets
'keyword "description" not in canonical order,expected "type", (See RFC 6020, Section 12)'

The leaf value, among other statements, clearly notes ";; these stmts can appear in any order" in RFC 6020, Section 12. The statement can mislead one to believe that the YANG is in error when in fact RFC 6020, Section 12 permits description appearing before type.

I tried to validate the following:

    leaf iref {
        type identityref {
            base x;
            base y ;
        }

I got back:

keyword 'base' already given for the second base line.

YANG 1.1 supports multiple base statements for identityrefs, so why do I get the error? Is it just not yet supported or is there some other problem here?

Hi,
the YANG validator used to produce the tree diagram as one of its outputs. Now, as I've found, it does not. Can you please kindly bring back "-f tree"?

Regards,
Greg

GET request on URL: https://yangcatalog.org/yangvalidator/api/versions should return json with versions of yangvalidator used, but currently return Server Error (500).

Current Django package installed is Django>=2.1.9. However, newer versions are available: 2.2.28, 3.2.14 or 4.0.6.
It might be necessary to check which version could be used and check whether existing functionality will no be broken by this update.

Yangvalidator times out if validating takes too long. Also there are several failures in yangvalidator logs

This is what the xym extraction output of RFC document number 7223 looks like:

   ERROR: '/var/yang/ietf/rfc/rfc7223.txt', Line 1584 - YANG module 'ex-ethernet' with no <CODE BEGINS> and not starting with 'example-'
   ERROR: '/var/yang/ietf/rfc/rfc7223.txt', Line 1666 - YANG module 'ex-ethernet-bonding' with no <CODE BEGINS> and not starting with 'example-'
   ERROR: '/var/yang/ietf/rfc/rfc7223.txt', Line 1713 - YANG module 'ex-vlan' with no <CODE BEGINS> and not starting with 'example-'

Among other things, it contains full path to cached file rfc7223.txt.
However, this information is not important for user, it only reveals where the files are stored on the server and also it also make response unnecessarily larger.

These paths are also found in the outputs of the individual validators and should be removed.

Hi, I'm not able to make the validator running locally. Probably due to missing some of the dependencies, but I'm out of ideas what next must be installed.

Running the main.py I'm getting the following error:

$ python --version
Python 2.7.6
$ sudo python ./bottle-yang-extractor-validator/main.py
Bottle v0.12.13 server starting up (using CherryPyServer())...
Listening on http://0.0.0.0:8080/
Hit Ctrl-C to quit.

Traceback (most recent call last):
  File "./main.py", line 253, in <module>
    run(server='cherrypy', host='0.0.0.0', port=port)
  File "/usr/local/lib/python2.7/dist-packages/bottle-0.12.13-py2.7.egg/bottle.py", line 3127, in run
    server.run(app)
  File "/usr/local/lib/python2.7/dist-packages/bottle-0.12.13-py2.7.egg/bottle.py", line 2787, in run
    from cherrypy import wsgiserver
ImportError: cannot import name wsgiserver

I have tried it on Ubuntu 14.04 and 16.04 and on openSUSE with the same results. In all cases the cherrypy-wsgiserver (which seems to be connected with the error according to its name) is installed in version 0.1dev.

as in title

new UI will expect json response instead of ready html document. Also take into consideration #9 an #39 issues since we will have new endpoints so we fix these issues as well

Noticed that today validation fails on YANG module that worked before:
[email protected]:9: error: module "ietf-inet-types" not found in search path
[email protected]:13: error: module "ietf-yang-types" not found in search path
[email protected]:17: error: module "ietf-key-chain" not found in search path
I've checked with BFD YANG and got similar failures:
Confdc Output
[email protected]:22: error: module 'ietf-routing' not found
[email protected]:23: error: module 'ietf-routing' not found
[email protected]:28: error: module 'ietf-key-chain' not found
[email protected]:13: error: module 'ietf-routing' not found
yanglint Validation
yanglint error: Unable to use search path (/var/tmp/yangmodules/extracted) - No such file or directory.

As a part of refactoring, please remove usage of .format() method for formatting strings.
Same is for joining strings using + character.
Instead, use f-strings for formatting strings.

The Pyang Validation section return the error:

Traceback (most recent call last):
pyang", line 444, in
run()
pyang", line 219, in run
repos = pyang.FileRepository(path, no_path_recurse=o.no_path_recurse)
init.py", line 408, in init
location = pip.locations.distutils_scheme('pyang')
AttributeError: 'module' object has no attribute 'locations'

To fix the same problem on my own version of pyang, I have commented out the following section in init.py.

'''
import pkgutil
if not pkgutil.find_loader('pip'):
return # abort search if pip is not installed

import pip
location = pip.locations.distutils_scheme('pyang')
self.dirs.append(os.path.join(location['data'],
'share','yang','modules'))
'''

use documents that we have in yangcatalog instead of using ietf.org/id which might take quiet a long time in some cases. If we don t have that document then use this url