Seems like some interop with using platform_slug_map from golden config and overloading a platform from that map in the dispatcher.

GC user has platform map of:

        "platform_slug_map": {"junos": "juniper_junos"},

And dispatcher point to netmiko juniper_junos

        "dispatcher_mapping": {
            "juniper_junos": "nornir_nautobot.plugins.tasks.dispatcher.juniper_junos.NautobotNornirDriver"
        }

Issue:

with dispatcher_mapping set to juniper_junos getting:

Unable to find the driver for check_connectivity for platform: junos

if setting the dispatcher_mapping to junos getting:

                                      ^
syntax error, expecting <command>.

While debugging this is because its pulling back command to run based on default key which is show run.

Workaround is to update:

https://github.com/nautobot/nornir-nautobot/blob/bb977910f9e711fe2190a71d61b641f277e73b80/nornir_nautobot/plugins/tasks/dispatcher/default.py#LL29C1-L37C2

and in this case add a junos key.

Add in auto publishing of new releases to Pypi

I am getting SSL: CERTIFICATE_VERIFY_FAILED, even though I am passing parameter

"ssl_verify": False

Code

`if name=="main":
nr = InitNornir(
inventory={
"plugin": "NautobotInventory",
"options": {
"nautobot_url": "https://10.81.161.100/",
"nautobot_token": "c6798e131be53ae38fd892fb6689144ca6d89c67",
"ssl_verify": False,
},
},
runner={
"plugin": "threaded",
"options": {'num_workers': 5}
},
logging={
"enabled": False
},
)

results=nr.run(task=helper_update)`

Output Error

`(.venv) tkdebnath@ubuntu:~/EIP$ python ip_helper.py
Traceback (most recent call last):
File "/home/tkdebnath/EIP/.venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 715, in urlopen
httplib_response = self._make_request(
File "/home/tkdebnath/EIP/.venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 404, in _make_request
self._validate_conn(conn)
File "/home/tkdebnath/EIP/.venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 1058, in validate_conn
conn.connect()
File "/home/tkdebnath/EIP/.venv/lib/python3.10/site-packages/urllib3/connection.py", line 419, in connect
self.sock = ssl_wrap_socket(
File "/home/tkdebnath/EIP/.venv/lib/python3.10/site-packages/urllib3/util/ssl.py", line 453, in ssl_wrap_socket
ssl_sock = ssl_wrap_socket_impl(sock, context, tls_in_tls)
File "/home/tkdebnath/EIP/.venv/lib/python3.10/site-packages/urllib3/util/ssl.py", line 495, in _ssl_wrap_socket_impl
return ssl_context.wrap_socket(sock)
File "/usr/lib/python3.10/ssl.py", line 513, in wrap_socket
return self.sslsocket_class._create(
File "/usr/lib/python3.10/ssl.py", line 1071, in _create
self.do_handshake()
File "/usr/lib/python3.10/ssl.py", line 1342, in do_handshake
self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate (_ssl.c:1007)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/tkdebnath/EIP/.venv/lib/python3.10/site-packages/requests/adapters.py", line 486, in send
resp = conn.urlopen(
File "/home/tkdebnath/EIP/.venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 799, in urlopen
retries = retries.increment(
File "/home/tkdebnath/EIP/.venv/lib/python3.10/site-packages/urllib3/util/retry.py", line 592, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='10.81.161.100', port=443): Max retries exceeded with url: /api/ (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate (_ssl.c:1007)')))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/tkdebnath/EIP/ip_helper.py", line 51, in
nr = InitNornir(
File "/home/tkdebnath/EIP/.venv/lib/python3.10/site-packages/nornir/init_nornir.py", line 72, in InitNornir
inventory=load_inventory(config),
File "/home/tkdebnath/EIP/.venv/lib/python3.10/site-packages/nornir/init_nornir.py", line 20, in load_inventory
inv = inventory_plugin(**config.inventory.options).load()
File "/home/tkdebnath/EIP/.venv/lib/python3.10/site-packages/nornir_nautobot/plugins/inventory/nautobot.py", line 148, in load
for device in self.devices:
File "/home/tkdebnath/EIP/.venv/lib/python3.10/site-packages/nornir_nautobot/plugins/inventory/nautobot.py", line 127, in devices
self._devices = self.pynautobot_obj.dcim.devices.all()
File "/home/tkdebnath/EIP/.venv/lib/python3.10/site-packages/nornir_nautobot/plugins/inventory/nautobot.py", line 110, in pynautobot_obj
self._pynautobot_obj = pynautobot.api(
File "/home/tkdebnath/EIP/.venv/lib/python3.10/site-packages/pynautobot/core/api.py", line 116, in init
self._validate_version()
File "/home/tkdebnath/EIP/.venv/lib/python3.10/site-packages/pynautobot/core/api.py", line 120, in _validate_version
api_version = self.version
File "/home/tkdebnath/EIP/.venv/lib/python3.10/site-packages/pynautobot/core/api.py", line 145, in version
).get_version()
File "/home/tkdebnath/EIP/.venv/lib/python3.10/site-packages/pynautobot/core/query.py", line 198, in get_version
req = self.http_session.get(
File "/home/tkdebnath/EIP/.venv/lib/python3.10/site-packages/requests/sessions.py", line 602, in get
return self.request("GET", url, **kwargs)
File "/home/tkdebnath/EIP/.venv/lib/python3.10/site-packages/requests/sessions.py", line 589, in request
resp = self.send(prep, **send_kwargs)
File "/home/tkdebnath/EIP/.venv/lib/python3.10/site-packages/requests/sessions.py", line 703, in send
r = adapter.send(request, **kwargs)
File "/home/tkdebnath/EIP/.venv/lib/python3.10/site-packages/requests/adapters.py", line 517, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='10.81.161.100', port=443): Max retries exceeded with url: /api/ (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate (_ssl.c:1007)')))
(.venv) tkdebnath@ubuntu:~/EIP$`

Modules installed

(.venv) tkdebnath@ubuntu:~/EIP$ pip freeze anyio==4.1.0 bcrypt==4.1.1 certifi==2023.11.17 cffi==1.16.0 charset-normalizer==3.3.2 colorama==0.4.6 cryptography==41.0.7 exceptiongroup==1.2.0 future==0.18.3 h11==0.14.0 httpcore==0.17.3 httpx==0.24.1 idna==3.6 Jinja2==3.1.2 junos-eznc==2.6.8 lxml==4.9.3 MarkupSafe==2.1.3 mypy-extensions==1.0.0 napalm==4.1.0 ncclient==0.6.13 netaddr==0.9.0 netmiko==4.3.0 netutils==1.6.0 nornir==3.4.1 nornir-jinja2==0.2.0 nornir-napalm==0.4.0 nornir-nautobot==3.0.0 nornir-netmiko==1.0.1 nornir-utils==0.2.0 ntc_templates==4.0.1 packaging==23.2 paramiko==3.3.1 pycparser==2.21 pyeapi==1.0.2 PyNaCl==1.5.0 pynautobot==2.0.2 pyparsing==3.1.1 pyserial==3.5 PyYAML==6.0.1 requests==2.31.0 ruamel.yaml==0.18.5 ruamel.yaml.clib==0.2.8 scp==0.14.5 six==1.16.0 sniffio==1.3.0 textfsm==1.1.3 transitions==0.9.0 ttp==0.9.5 ttp-templates==0.3.5 typing_extensions==4.9.0 urllib3==1.26.18 yamlordereddictloader==0.4.2

Please update httpx to latest version.

Multiple users have had issues where the import statements need to be updated in dispatcher/default.py.

From:

from netmiko.ssh_exception import NetmikoAuthenticationException, NetmikoTimeoutException

To:

from netmiko import NetmikoAuthenticationException, NetmikoTimeoutException

The nautobot-ansible plugin allows the option of overwriting the inventory host name with a virtual chassis name:
https://nautobot-ansible.readthedocs.io/en/latest/plugins/inventory_inventory.html#parameter-virtual_chassis_name, all of the host vars are from the master member.

When it comes to limiting within ansible you're then able to limit of the virtual chassis name.

I'm looking to do the same thing but with nornir-nautobot.

Currently nornir-nautobot would return only devices and not virtual-chassis but I'm looking for the ability to pass via cli something like --hostname <virtualchassis> then be able to use an F filter to filter on the 'name' attribute to filter the inventory to just those devices matching the hostname(s) passed in via cli.

Currently pyproject.toml only supports netutils versions <1. Update pyproject.toml and poetry.lock to support the latest netutils

pynautobot has the capability to disable threading. It is disabled by default. This should be exposed as an option and accounted for.

Support a new option passed in:

enable_threading

Currently when the dispatched fails, the operator is provided with an error based on f"Unable to find the driver for {method} for platform: {task.host.platform}"). It would be good to add a link into that error message to point to a website of what was supported from a platform and method perspective, similar to the napalm getters.

Not able to install package in edit mode due to non PEP-517 compliant build-system config in pyproject.toml. Traceback when attempting this:

pip install -e plugins/ah-nornir-nautobot

Obtaining file:///Users/anthonyhouse/git_repos/nautobot-local-env/plugins/ah-nornir-nautobot
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
ERROR: Project file:///Users/anthonyhouse/git_repos/nautobot-local-env/plugins/ah-nornir-nautobot has a 'pyproject.toml' and its build backend is missing the 'build_editable' hook. Since it does not have a 'setup.py' nor a 'setup.cfg', it cannot be installed in editable mode. Consider using a build backend that supports PEP 660.

There should be able to build in tests for functions pretty easily for REST API driven systems.

Should be:
nornir_nautobot.plugins.tasks.dispatcher.cisco_ios_xr.NautobotNornirDriver

Hi all.

In the Build Inventory method of nornir-nautobot we can see:

host_platform=device.platform.slug

But looking at Nautobot 1.0, it is something more like:

host_platform=device.device_type.manufacturer.slug

That is working fine...

Thanks
Regards

Add ability to add options for groups, in a similar fashion to what the Ansible inventory module will support.

Configuration

options: {
  "group_by": ["tags", "sites"]
}

Underneath considerations

• Should have a finite list of options to group by that are known to work
• Unit tests
• Verify that the group_by option that comes in will work

Right now if you go into > mode on cisco_ios and you don't have permissions to run show run it isn't going to try the DEVICE_SECRET and go into enable mode to get the config.

The job itself shows successful but the backup itself in Git is saved as:

                      ^
% Invalid input detected at '^' marker.

Hallo!

if revert_in is used in next method :

napalm_confirm_commit needs to be used to actually confirm the commit:

from nornir_napalm.plugins.tasks import napalm_confirm_commit
if revert_in is not None:
    confirm_result = task.run(task= napalm_confirm_commit)

otherwise, the system will rollback to the backup config every time.

It currently would appear that devices do not allow for the secret password to actually do anything.

When working with the generate_config function the current behavior has any Exceptions from Jinja2 being masked by raising a NornirNautobotException with log message. This means that any logging that is done in a Job is unable to see the full traceback. If we instead re-raised the same Exception we'd have access to the traceback and could log it. The traceback is essential with config generation as it allows quick identification of where the error lies. For example:

The full traceback is this:

[ERROR/ForkPoolWorker-6] Host 'jcy-spine-01.infra.ntc.com': task 'template_file' failed with traceback:
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/nornir/core/task.py", line 99, in start
    r = self.task(self, **self.params)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/nornir_jinja2/plugins/tasks/template_file.py", line 43, in template_file
    text = t.render(host=task.host, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/jinja2/environment.py", line 1301, in render
    self.environment.handle_exception()
  File "/usr/local/lib/python3.11/site-packages/jinja2/environment.py", line 936, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File "/opt/nautobot/git/templates/cisco_nxos.j2", line 103, in top-level template code
    {% include './nxos/vlans.j2' %}
    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/nautobot/git/templates/nxos/vlans.j2", line 1, in top-level template code
    {% for vlan in site["vlans"] %}
    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/jinja2/sandbox.py", line 303, in getitem
    return obj[argument]
           ~~~^^^^^^^^^^
jinja2.exceptions.UndefinedError: 'site' is undefined

but the Job logs only show E1010: There was a jinja2.exceptions.UndefinedError error: 'site' is undefined. You have to look at the STDERR logs to find the traceback. It'd be much more useful if we could log the traceback as that's where the important information is.

Currently the nornir-netmiko version is restricted to >=0.0.0 <0.1.0 while the latest version available is 1.0.0. I can raise a PR to update the pyproject.toml and the lock file if there is an agreement.

The get_config methods found in the dispatcher does not close the connection per host. In golden config connections for this task are not closed until after the task has run for every host.
This can cause issues such as:

• HIgh CPU usage
• HIgh Memory usage
• Reaching the limit if the ulimit parameter is set low.

Please update dependencies for all packages but specifically nornir-jinja2

Looking to bump netmiko to 4.x support

Check compatibility with Nautobot v2

Problem occure when we have more then one line to substitute from the running config.
what happends is that for every extra line we have defined in substitute-line we also get an extra copy of the config-line.

example issue when having 3 lines defined to match against:

Building configuration...
Building configuration...
Building configuration...



Current configuration : 76809 bytes
Current configuration : 76809 bytes
Current configuration : 76809 bytes

suggestion:

       # Loop through the config lines
        for line in config.split("\n"):
            # Loop through the replacement list on each line in the config
            match = False
            for item in substitute_lines:
                if re.match(pattern=fr"{item['regex_search']}", string=line):
                    line = line.replace(
                        re.match(pattern=fr"{item['regex_search']}", string=line).groups()[0], item["regex_replacement"]
                    )
                    match = True
                if match is True:
                    new_config += line.rstrip() + "\n"
                    break
            if match is False:
                new_config += line.rstrip() + "\n"
        config = new_config

It's impossible for now to connect to a device through a proxy SOCKS or any bastion server, would be possible to add this feature ?

Currently when running the default get_config we need to send in a file path to backup_file so it can save the config to a file:

If we want to get the configuration but not save it we should be able to send in an empty string to disable this functionality.

Add flag to unpack the pynautobot object into a dictionary. Should default to True to unpack.

Some ios device have slightly diffrent outputs and cause this to not get caught and handled.
Example

                      ^
% Invalid input detected at '^' marker.

although the API response includes the "platform" the variable task.host.platform is not set:

{ "count": 7, "next": null, "previous": null, "results": [ { ... "platform": { ... "name": "cisco_ios", "slug": "cisco_ios" }, ...

Seeking for a solution to:

a) store platform-dependent API endpoints (equivalent or extends RUN_COMMAND_MAPPING constant in default dispatcher).
b) provides user the ability to edit on a per platform basis (add, remove, override) default platform API endpoints.

this is linked to PR:: #79

PR solution (mikrotik routeros only):

ROUTEROS_API_ENDPOINTS = [
    "/system/identity",
    "/interface",
    "/ip/address",
    "/system/ntp/client",
    "/ip/dns",
    "/snmp/community",
    "/system/logging/action",
]

RUN_COMMAND_MAPPING = {
    "default": "show run",
    "cisco_nxos": "show run",
    "cisco_ios": "show run",
    "cisco_xr": "show run",
    "juniper_junos": "show configuration | display set",
    "arista_eos": "show run",
    "ruckus_fastiron": "show running-config",
    "mikrotik_routeros_api": ROUTEROS_API_ENDPOINTS,
}

Potential solution might be to move this to a class attribute in the dispatcher itself (proposed by @itdependsnetworks )

Nornir allows you to set defaults for defaults for Host attributes, like username and password.
This currently doesn't seem to work when using nornir_nautobot.

Consider the following code snippet

from nornir import InitNornir

nr = InitNornir(
    inventory={
        "plugin": "NautobotInventory",
        "options": {
            "nautobot_url": "https://myinstance.of.nautobot.tld",
            "nautobot_token": "api_token",
        },
    },
)

nr.inventory.defaults.username = "username"

The expectation would be that every host in the inventory has it's username attribute set with the value "username", if no explicit value was set for that particular host.

>>> nr.inventory.hosts["a_host"].username
"username"

However the actual result is

>>> nr.inventory.hosts["a_host"].username
>>> nr.inventory.hosts["a_host"].username is None
True

For this to work properly, the inventory plugin would have to instantiate a Hosts object with a defaults attribute, which doesn't seem to happen right now.

Some versions of cisco_ios(xe) don't have ERROR: in the output. This can cause the backup configuration to actually be saved as

               ^
% Invalid input detected at '^' marker.

Hello,

I wasn't sure under which Github repository to leave this issue so if I should open it under the nornir_netmiko library instead let me know.

Issue

The nornir_nautobot library allows us to map non-standard platform slugs to the correct [Nornir Dispatcher class].(https://docs.nautobot.com/projects/plugin-nornir/en/latest/user/app_feature_dispatcher/#configuring-the-dispatcher)

This is very convenient, and I was attempting to take advantage of this but noticed my Nautobot Golden Config backup jobs were still failing. Here is the code trace I've done and what I believe is the root cause of the failure:

1. nornir_nautobot selects the correct dispatched based on the extra configurations we make, and eventually ends up in the get_config method of the NetmikoNautobotNornirDriver class:
   https://github.com/nautobot/nornir-nautobot/blob/develop/nornir_nautobot/plugins/tasks/dispatcher/default.py
2. This method calls the netmiko_send_command task.
3. This then attempts to establish a Netmiko connection object:
   https://github.com/ktbyers/nornir_netmiko/blob/develop/nornir_netmiko/tasks/netmiko_send_command.py
   https://github.com/ktbyers/nornir_netmiko/blob/develop/nornir_netmiko/connections/netmiko.py
4. This connection object attempts to use the platform string to map to the correct Netmiko device_type:

napalm_to_netmiko_map = {
    "ios": "cisco_ios",
    "nxos": "cisco_nxos",
    "nxos_ssh": "cisco_nxos",
    "eos": "arista_eos",
    "junos": "juniper_junos",
    "iosxr": "cisco_xr",
}

However, because we are using a non-standard platform string this mapping fails and throws an exception.

The Ask

Would it be possible to allow us to either also configure the napalm_to_netmiko_map to include our non-standard platform strings, or have some way of passing an overriding platform value when calling the netmiko_send_command task and establishing the Netmiko connection.

Another possibility may be to use the Napalm driver instead, but I also believe there modifications would be necessary to pass a configured platform slug.

I am of course open to any other suggestions or ideas you may have for working around this issue and please let me know if you'd like more information.