napalm-automation-community / napalm-sros Goto Github PK

The last commit broke few thing

• I agree that's a correct approach of using optional args (passing the optional args in constructor -> that can change behavior (e.g. format) of other functions for that driver object)
• But with current implementation the 'optional_args' can't be used to alter format in get_config and json_format in compare_config
• Also I believe now when you do following calls then it fails. Calling get_config make compare_config failing and vice versa.

device.get_config()
device.compare_config()

The reason is that self.optional_args should be assigned to only in constructor and in read only in the other functions.

The current implementation is that in constructor the None is always assigned to self.optional_args -> then get_config checks self.optional_args is None, in this case it is so it assigns default value for 'format' to it -> when calling compare_config the self.optional_args is not None but it doesn't have 'json_format' key either, so it fails.

A way how to implement this could be to have in the constructor

self.optional_args = optional_args

and then e.g. in compare_config having

json_format = self.optional_args.get("json_format", False)

and to use json_format variable in checks if to return diff in json format or not

We're using this napalms driver to replace config file (we're using xml format)

driver = napalm.get_network_driver('sros’)
driver.load_replace_candidate(filename=candidate_conf_path)

but the switch config is locked from CLI, so we get

Traceback (most recent call last):
  File "/Users/riccardo/Projects/NCA/pysros/pysros/command_line.py", line 189, in <module>
    main()
  File "/Users/riccardo/Projects/NCA/pysros/pysros/command_line.py", line 16, in wrapper
    func()
  File "/Users/riccardo/Projects/NCA/pysros/pysros/command_line.py", line 177, in main
    device.load_replace_candidate(filename=candidate_conf_path)
  File "/Users/riccardo/virtualenvs/pysros/lib/python3.6/site-packages/napalm_sros/sros.py", line 471, in load_replace_candidate
    self._lock_config()
  File "/Users/riccardo/virtualenvs/pysros/lib/python3.6/site-packages/napalm_sros/sros.py", line 199, in _lock_config
    self.conn.lock()
  File "/Users/riccardo/virtualenvs/pysros/lib/python3.6/site-packages/ncclient/manager.py", line 231, in execute
    huge_tree=self._huge_tree).request(*args, **kwds)
  File "/Users/riccardo/virtualenvs/pysros/lib/python3.6/site-packages/ncclient/operations/lock.py", line 35, in request
    return self._request(node)
  File "/Users/riccardo/virtualenvs/pysros/lib/python3.6/site-packages/ncclient/operations/rpc.py", line 360, in _request
    raise self._reply.error
ncclient.operations.rpc.RPCError: 
            MINOR: MGMT_CORE #2052: Exclusive datastore access unavailable - the MD-CLI has exclusive lock on configuration

but the function hangs there instead of returning back to python prompt.
Please check why this call hangs.

Also in the driver it seems to be ready to raise "standard" napalm exceptions https://github.com/napalm-automation-community/napalm-sros/blob/master/napalm_sros/sros.py#L500 , but this doesn't seem to be working in this case as well.

When doing compare_config() with configs in XML format, the compare_config doesn't return human readable string.

From napalm docs:

A string showing the difference between the running configuration and the candidate configuration.
https://napalm.readthedocs.io/en/latest/base.html#napalm.base.base.NetworkDriver.compare_config

Currently the compare_config() returns dictionary and the output contains the xml namespaces, which isn't very human friendly and readable.

test_get_interfaces passes om CI/CD but does not pass on local system due to differences in date and time operation on different operating systems

the code in get_interfaces method in sros.py:

ifd["last_flapped"] = (
  datetime.datetime.strptime(
      flap_time, "%Y-%m-%dT%H:%M:%S.%fZ"
  ).timestamp()
  if flap_time != ""
  else -1.0
)

The init_logging() adds logging to a file in current working directory.

But because the sros napalm driver is loaded "dynamically" from napalm library (get_network_driver() function), it's very hard to disable this logging to file and use different logging setup (e.g. to syslog, or even no logging).

Please add possibility to disable this logging file handler and not changing logging module setting.

discard_config() fails when the candidate config is the same as the running config.

I'm using configs in XML format and doing

>>> device.close()
>>> device.open()
>>> device.load_merge_candidate('./tests/payloads/filter200-swap.xml')
>>> device.compare_config()
''
>>> device.discard_config()
Traceback (most recent call last):
File "", line 1, in
File "/home/marek/Projects/nokia-napalm-tests/.tox/py3-test/lib/python3.8/site-packages/napalm_sros/sros.py", line 272, in discard_config
self._unlock_config()
File "/home/marek/Projects/nokia-napalm-tests/.tox/py3-test/lib/python3.8/site-packages/napalm_sros/sros.py", line 206, in _unlock_config
self.conn.unlock()
File "/home/marek/Projects/nokia-napalm-tests/.tox/py3-test/lib/python3.8/site-packages/ncclient/manager.py", line 239, in execute
return cls(self._session,
File "/home/marek/Projects/nokia-napalm-tests/.tox/py3-test/lib/python3.8/site-packages/ncclient/operations/lock.py", line 49, in request
return self._request(node)
File "/home/marek/Projects/nokia-napalm-tests/.tox/py3-test/lib/python3.8/site-packages/ncclient/operations/rpc.py", line 367, in _request
raise self._reply.error
ncclient.operations.rpc.RPCError:
MINOR: NETCONF #2052: Current session does not hold the lock

The same happens with load_replace_candidate. But if the candidate config differs the discard_config() works fine.

I am trying to access information on our new NOKIA 7750 routers running SROS 23.3.R1 with napalm-sros and I'm running into a problem. Any function that calls XML parsing with the namespace returns the error "an unexpected namespace is present".

Here is my code (This code accesses Juniper and NOKIA equipment):

for device, details in devices.items():
    driver = get_network_driver(details['system'])
    print(driver)
    if details['system'] == 'JunOS':
        connection = driver(username=JUNIPER_USERNAME, password=JUNIPER_PASSWORD, hostname=details['ip'])
    else:
        connection = driver(username=NOKIA_USERNAME, password=NOKIA_PASSWORD, hostname=details['ip'])
    try:
        connection.open()
        if not connection.is_alive():
            print('Unable to log into device')
            print(connection)
        print("interfaces")
        interfaces = connection.get_interfaces()
        pprint(interfaces)
        print('Interfaces with IP')
        ips = connection.get_interfaces_ip()
        pprint(ips)
        print('Network instances')
        instances = connection.get_network_instances()
        pprint(instances)
        print('facts')
        pprint(connection.get_facts())
        print('LLDP neighbors')
        pprint(connection.get_lldp_neighbors_detail())
        print('optics')
        pprint(connection.get_optics())
        print('config')
        pprint(connection.get_config())
   finally:
        connection.close()

And this is the following output:

<class 'napalm_sros.sros.NokiaSROSDriver'>
interfaces
Error in method get interfaces : 
            An unexpected namespace is present.
        
None
Interfaces with IP
Error in method get interfaces ip : 
            An unexpected namespace is present.
        
None
Network instances
Error in method get network instances : 
            An unexpected namespace is present.
        
None
facts
Error in method get facts : 
            An unexpected namespace is present.
        
None
LLDP neighbors
Error in method get lldp neighbors detail : 
            An unexpected namespace is present.
        
None
optics
Error in method get optics : 
            An unexpected namespace is present.
        
None
config
{'candidate': '<configure xmlns="urn:nokia.com:sros:ns:yang:sr:conf" '
              'xmlns:nokia-attr="urn:nokia.com:sros:ns:yang:sr:attributes" '
              'xmlns:nc="urn:ietf:params:xml:ns:netconf:base:1.0">\n'
              (insert config here)
              '    ',
 'running': '<configure xmlns="urn:nokia.com:sros:ns:yang:sr:conf" '
            'xmlns:nokia-attr="urn:nokia.com:sros:ns:yang:sr:attributes" '
            'xmlns:nc="urn:ietf:params:xml:ns:netconf:base:1.0">\n'
           (insert config here)
            '    ',
 'startup': '<configure xmlns="urn:nokia.com:sros:ns:yang:sr:conf" '
            'xmlns:nokia-attr="urn:nokia.com:sros:ns:yang:sr:attributes" '
            'xmlns:nc="urn:ietf:params:xml:ns:netconf:base:1.0">\n'
           (insert config here)
            '    '}

When I run show version on the device, I see this:

# show version
TiMOS-C-23.3.R1 cpm/hops64 Nokia 7750 SR Copyright (c) 2000-2023 Nokia.
All rights reserved. All use subject to applicable license agreements.
Built on (time) by builder in (location)

Calling close() method of the driver doesn't close the SSH connection to the device.

This is causing the configuration is still locked by the "napalm connection" so when connecting to CLI manually I can't run configure global

MINOR: MGMT_CORE #2051: Global datastore access unavailable - the MD-CLI has exclusive lock on configuration

To fix this issue I needed to run device.conn_ssh.get_transport().close() manually.

In 21.10 it's power-shelf: https://github.com/nokia/7x50_YangModels/blob/master/latest_sros_21.10/nokia-state-chassis.yang#L1460
however for some reason on VSR there is only (emulated) power-supply

Consequently, get_environment fails completely, even though other information like CPU/memory utilization is available

Proposing to create either creating labels(git tags) or release branchs.

Impact

1. Possiblity of Version pinning
2. No unexpected changes in different envs when installing from requirements.txt
3. General benifit of easier tracking of issues and upgrades.

Any guidance on how to integrate this into the bigger Napalm project? What to put in _SUPPORTED_DRIVERS.py? How to call it using the napalm command line utility? I tried, but seems like there is some inconsistency in the package naming: napalm_sros, napalm-sros, sros? Thanks

When deploying a new router I stumbled upon the issue that a config merge error is not reported back via the Napalm SROS module in Peering Manager, but when I do the config change manually Incuding commit, I do get these messages. Is this expected behaviour ? I would like to get feedback If my commit is not working.

Config merge from Peering manager via Napalm: (No error message is reported back, but I would expecet the message of policy-variable definition-missing )

2023-10-28 08:42:57,825 session.py _dispatch_message line 78 DEBUG:  [host sros02 session-id 44] dispatching message to listener: <ncclient.operations.rpc.RPCReplyListener object at 0x7fccae25bf70>
2023-10-28 08:42:57,826 rpc.py callback line 255 DEBUG:  [host sros02 session-id 44] Delivering to <ncclient.operations.session.CloseSession object at 0x7fccae1f7220>
2023-10-28 08:42:57,826 session.py _dispatch_message line 78 DEBUG:  [host sros02 session-id 44] dispatching message to listener: <ncclient.transport.session.NotificationHandler object at 0x7fccae25bcd0>
2023-10-28 08:42:57,826 parser.py _parse11 line 233 DEBUG:  [host sros02 session-id 44] _parse11: saving back rest of message after 44 bytes, original size 44
2023-10-28 08:42:57,827 parser.py _parse11 line 240 DEBUG:  [host sros02 session-id 44] _parse11: ending
2023-10-28 08:42:57,827 ssh.py run line 529 DEBUG:  [host sros02 session-id 44] Broke out of main loop, error=SessionCloseError('Unexpected session close')
2023-10-28 08:42:57,827 session.py _dispatch_error line 85 DEBUG:  [host sros02 session-id 44] dispatching error to <ncclient.operations.rpc.RPCReplyListener object at 0x7fccae25bf70>
2023-10-28 08:42:57,828 session.py _dispatch_error line 85 DEBUG:  [host sros02 session-id 44] dispatching error to <ncclient.transport.session.NotificationHandler object at 0x7fccae25bcd0>
2023-10-28 08:42:57,829 __init__.py close_napalm_device line 1248 DEBUG:  closed connection with sros02
2023-10-28 08:42:57,829 rpc.py _request line 352 INFO:  [host sros02 session-id 44] Requesting 'CloseSession'
2023-10-28 08:42:57,830 jobs.py log line 182 INFO:  Configuration installed.
2023-10-28 08:42:57,843 worker.py perform_job line 1463 INFO:  default: Job OK (7e41eb58-e0ed-4b3f-81a4-788b3ecda791)
2023-10-28 08:42:57,843 worker.py perform_job line 1472 INFO:  Result is kept for 500 seconds

Manual config change on Router: (There were some policy-variable definition-missing , clear error output)

A:admin@sros2# commit
  MINOR: MGMT_CORE #224: configure policy-options policy-statement "PMGR-PRIVATE-PEERING-IN-V4" entry 10 from policy - Entry does not exist - subpol policy-statement name "PEER-IN-V4", policy-variable definition-missing or out-of-range "@COMMUNITY@" - configure policy-options policy-statement "PMGR-PRIVATE-PEERING-IN-V4" entry 10
  MINOR: MGMT_CORE #224: configure policy-options policy-statement "PMGR-TRANSIT-IN-V4" entry 10 from policy - Entry does not exist - subpol policy-statement name "PEER-IN-V4", policy-variable definition-missing or out-of-range "@COMMUNITY@" - configure policy-options policy-statement "PMGR-TRANSIT-IN-V4" entry 10
  MINOR: MGMT_CORE #224: configure policy-options policy-statement "PMGR-TRANSIT-LGI-IN-V4" entry 10 from policy - Entry does not exist - subpol policy-statement name "PEER-IN-V4", policy-variable definition-missing or out-of-range "@COMMUNITY@" - configure policy-options policy-statement "PMGR-TRANSIT-LGI-IN-V4" entry 10

I had an issue with the latest updated 4.6.4 version of LXML, import etree doesn't work anymore for some reason. Would suggest to make this an exact requirement (==4.6.2) or fix the import issue (new version has some CVE security fixes)

get_config() returns persistent-indices section

persistent-indices {\n    description "Persistent system indices are loaded at boot time and cannot be modified."\n}\n'

at the end of the get_config()['running'] . So when I try to replace the obtained running configuration back if fails because
MGMT_CORE #2201: Unknown element - 'persistent-indices'
(I've used #8 to get reason of the replace failure)

Inspecting the sros.py code I can see that in multiple places the exception is caught for logging the exception into log. But the exception is not re-raised, so it's logged but then the code continues in processing.

This is quite misleading and hard to properly integrate the sros driver into larger projects (the same applies to having prints in the driver which can't be "suppressed" from the application - the logging is configurable, the prints not).
So I can see issues like

Error in opening netconf connection : Could not open socket to edge3-uat.linx.net:830
Error in method get config : 'NoneType' object has no attribute 'get_config'
Traceback (most recent call last):
  File "/home/marek/.local/bin/pysros", line 33, in <module>
    sys.exit(load_entry_point('pysros==0.0.4', 'console_scripts', 'pysros')())
  File "/home/marek/.local/lib/python3.8/site-packages/pysros/command_line.py", line 17, in wrapper
    func()
  File "/home/marek/.local/lib/python3.8/site-packages/pysros/command_line.py", line 170, in main
    running.write(result['running'])
TypeError: 'NoneType' object is not subscriptable

The reason for the failure is that the device is unreachable, but TypeError: 'NoneType' object is not subscriptable is completely unrelated and very misleading.
That's even harder to debug when using the sros driver with projects like napalm-ansible because it doesn't show the printed command but just fails with 'NoneType' object has no attribute 'edit_config' :-)

Please re-raise the exceptions as that's the standard behaviour and the "application" should handle those issues not to have them suppressed in "library".

Command get_mac_address_table return error with incorrect path text parser (Centos 8, python 3.9.1)

Traceback (most recent call last):
File "/opt/venv/scripts/napalm_test.py", line 23, in
sros_out3=device.get_mac_address_table()
File "/opt/venv/lib/python3.9/site-packages/napalm_sros-0.1.0-py3.9.egg/napalm_sros/sros.py", line 2165, in get_mac_address_table
output_list = parse_with_textfsm(template, buff)
File "/opt/venv/lib/python3.9/site-packages/napalm_sros-0.1.0-py3.9.egg/napalm_sros/utils/parse_output_to_dict.py", line 11, in parse_with_textfsm
with open(os.path.join(os.path.dirname(os.path.realpath(file)), template), "r") as template_file:
FileNotFoundError: [Errno 2] No such file or directory: '/opt/venv/lib/python3.9/site-packages/napalm_sros-0.1.0-py3.9.egg/napalm_sros/utils/textfsm_templates\nokia_sros_show_service_fdb_mac.tpl'

Line 2162 - 2165

    cmd = "show service fdb-mac"
    buff = self._perform_cli_commands(["environment more false", cmd])
    template = "textfsm_templates\\nokia_sros_show_service_fdb_mac.tpl"
    output_list = parse_with_textfsm(template, buff)

Replacing '//' with '' resolves the issue, however it should be adapted automatically.

In nc_filters.py a list of VPLS interfaces is queried. However there is no such element possible under a VPLS service, only SAP or EVPN attachments can be done. This results in:

print(device.get_network_instances(name=""))
ncclient.operations.rpc.RPCError:
MINOR: MGMT_CORE #2201: Unknown element

When you remove the VPLS service from filters below and only query base router and VPRN hierarchy this works fine.

Related code block listed below.

The generate timestamp in the get_interface unittest does not match the mock data when running on Windows devices. It does match when running on MacOS, GitHub and Linux.

When using this driver to do configuration changes (replace/merge) with xml format, the validity of XML doesn't seem to be checked.

So in case the xml file isn't valid (e.g. missing closing tag) , the driver proceeds with the file, prints no error and hangs. So there is no hint about the provided config file is syntactically incorrect.
The issue about function call hanging can be related to #27 .

When the configuration is locked it is not possible to create a diff.
Error in locking the session caused exception: MINOR: MGMT_CORE #2052: Exclusive datastore access unavailable - model-driven interface editing global candidate

Ideally since in a diff no change in the config is performed the diff should work even when the config is locked.

Running 'get_config()' twice fails to get the running config because of ssh connection in incorrect mode.

Running:

>>> device.open()
>>> device.get_config()
>>> config = device.get_config()
>>> config['running']
"admin show configuration | no-more\n                          ^^^^^\nMINOR: MGMT_CORE #2201: Unknown element - 'admin'\n"

config['candidate'] seems to contain correct config.

Description:

The documentation should specify that the current driver code is using SSH for the get_config() method and not NETCONF.

Install.md should include "console true" in order to make it work:

user "netconf" {
        password "<hash>"
        access {
            console true
            netconf true
        }
        console {
            member ["administrative"]
        }
    }

See #57 for a fix

Biggest issue is that some peers are omitted completely. For those that are included, 13 out of 35 attributes were incorrect

Steps to reproduce

Install napalm-sros with pip

pip install git+https://github.com/napalm-automation-community/napalm-sros.git

Current behaviour

Setup.py has textfsm pinned to incorrect version, https://github.com/napalm-automation-community/napalm-sros/blob/master/setup.py#L24.

pip install git+https://github.com/napalm-automation-community/napalm-sros.git

ERROR: Could not find a version that satisfies the requirement textfsm>=1.2.0 (from napalm-sros) (from versions: 0.3.2, 0.4.0, 0.4.1, 1.1.0, 1.1.2, 1.1.3)
ERROR: No matching distribution found for textfsm>=1.2.0

The change was done on this commit, 88c0ccb

The version is set correct on the requirements.txt file, 88c0ccb#diff-4d7c51b1efe9043e44439a949dfd92e5827321b34082903477fd04876edb7552

Expected behaviour

Napalm-sros would be installed

When showing the diff for a port admin-state change on an SR-7s, the diff does shows port numbers which cannot be correlated to any port:

[
    "change",
    [
        "configure",
        "port",
        0,
        "admin-state"
    ],
    [
        "enable",
        "disable"
    ]
]

Instead the diff should show the actual port name:

[
    "change",
    [
        "configure",
        "port",
        "1/1/c1",
        "admin-state"
    ],
    [
        "enable",
        "disable"
    ]
]

diff.json.txt

Steps to reproduce

While napalm-sros 1.0.1 works with napalm 3.4.1/netmiko 3.4.0/textfsm 1.1.3, it doesn't work with Napalm 4.0.0

Install napalm-sros with pip with napalm 4.0.0

pip install git+https://github.com/napalm-automation-community/napalm-sros.git

Current behaviour

Napalm 4.0.0 has pinned netmiko version

netmiko>=4.1.0

https://github.com/napalm-automation/napalm/blob/develop/requirements.txt#L11

and Netmiko 4.1.2 has pinned textfsm version

textfsm==1.1.2

https://github.com/ktbyers/netmiko/blob/develop/setup.py#L55

Expected behaviour

Napalm-sros would be installed

After I load a replace candidate the compare_config() returns for me non-printable characters (probably result of 'dots loading animation' in CLI)

>>> device.compare_config()
'...\x08\x08\x08   \x08\x08\x08...\x08\x08\x08   \x08\x08\x08...\x08\x08\x08   \x08\x08\x08'

(note: the loaded candidate config should change no config in running config)

I've configured Nokia 7750 SR-2s device following the installation guide from this repo.

I can connect to the device and I can get the running config, but it fails when I try to load the obtained config back.

>>> device.open()
>>> config = device.get_config()['running']
>>> device.load_replace_candidate(config=config)
Traceback (most recent call last):
  File "/srv/conda/envs/python3.6/lib/python3.6/site-packages/napalm_sros/sros.py", line 376, in load_replace_candidate
    raise ReplaceConfigException()
napalm.base.exceptions.ReplaceConfigException

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/srv/conda/envs/python3.6/lib/python3.6/site-packages/napalm_sros/sros.py", line 378, in load_replace_candidate
    raise ReplaceConfigException(rex)
napalm.base.exceptions.ReplaceConfigException

Note: the obtained running config is a string with no new lines.

Can this failure be related to any device config setting? Or how to fix this?

get_config() returns MOTD as part of the running config

>>> config = device.get_config()
>>> config['running']
'Access restricted to LINX!\nconfigure {\n    card 1 {\n       ...

Hi,
please follow napalm "interface" so this driver for SROS is interchangeable with other vendors and compatible with wrappers like napalm-ansible.

It's great that now it's possible to do more performant diffs with xml format, but first issue I've seen is that the compare_config() signature is different and doing the compare with config in function arguments is not compatible with napalm and other tools.

This repo seems to have unit tests for comparing signature of the driver with upstream base driver. But this probably wasn't run

E AssertionError: Some methods vary.
E dict_keys(['compare_config', 'get_config', 'ping', 'traceroute'])

Also xmltodict and dictdiffer dependencies are missing in the requirements files.