enms-automation / enms Goto Github PK

When a filter based on nodes and links properties is created, it should be saved in the database to be reused afterwards, without having to reenter manually all regex / property values.

• listbox to let the user choose a filter in the Object filtering webpage.

Create a special type of script using Netmiko ConnectHandler to transfer one or several files, to or from the device.

ssh_conn = ConnectHandler(**net_device)

source_file = 'test1.txt'
dest_file = 'test1.txt'
file_system = 'flash:'
direction = 'put'

transfer_dict = file_transfer(ssh_conn, source_file=source_file, dest_file=dest_file,
                              file_system=file_system, direction=direction,
                              overwrite_file=False, disable_md5=False)
print(transfer_dict)

Return dict value:

{ 'file_exists': True, 
  'file_transferred': True, 
  'file_verified': True }

• use netbox API to pull the data and store in the DB
  https://netbox.readthedocs.io/en/latest/api/overview/

Hi,when I excute python app.py ,it occured a error.
C:\python\net>python app.py
File "app.py", line 7

^
SyntaxError: invalid syntax
why.

Right now, there is no way to edit an object other than deleting it and recreating it.
To be implemented: user-friendly way to edit an object

When nodes are colocated, we cannot distinguish them from one another with classic leaflet.
We need to use leaflet markercluster https://github.com/Leaflet/Leaflet.markercluster for colocated nodes to be displayed as a cluster, like on Google Earth.

it would be nice if it can do Radius authentication too.

Add Dockerfile

When creating a script, a YAML file can be provided by the user.

If the script is a Jinja2-based template, before sending the script to a group of devices, a Flask form should be generated based on this YAML file.

The default values of this form should be the ones provided in the YAML Flask.

This will provide a way for the user to change the values of the YAML file "on the fly" before sending the script (thoses values might differ depending on the target devices).

Debug docs (https://github.com/afourmy/eNMS/blob/master/docs/base/installation.rst) need to be updated to include the ENMS_CONFIG_MODE environment variable created as part of e206190.

I'm happy to raise a PR for this if you'd like. Do you write the RST files by hand, or do they pull their info from code?

It should be possible to send a script not only from the netmiko / napalm webpage, but also directly from the geographical and logical views:

• implement a selection box for the user to be able to select multiple devices at once (left-click button)
• implement a bootstrap modal for the user to send a netmiko / napalm script directly from the JavaScript view.

In the end, when this method is fully working for both leaflet and vis, this could replace entirely the automation blueprint. (=> merging of the automation and view blueprints)

In the scripts menu, currently simple scripts is made with name + content ; so it's should be to make scripts with vendor / os field (like cisco / ios, ios-xr / asa ; junipe junos ..). From that, we can easy to map for device - scripts

Right now, the SSH connection to a device only works on Windows as it uses subprocess.Popen to call Putty.exe. With Unix, the ssh command must be called with subprocess.Popen or subprocess.call.

If eNMS is installed on a jump server, the SSH connection should also work if X11 forwarding / Xming is enabled on the SSH connection to the jump server.

Introduce the concept of "variable script".

A script that contains both plain text and variables.
Two types of variables:
$variable -> upon sending the script, the user will be asked to fill in a form that contains textfields for all $variable input of the script.
@node.property -> upon sending the script, eNMS will automatically look into the target nodes property, and replace @node.property with the appropriate value.

When selecting multiples devices (and multiple getters for the napalm getters page), the napalm connections are processed as a simple python loop.
Improvement: use multithreading or multiprocessing (or both and let the user choose which option he prefers) to start all napalm connections (resp all netmiko ConnectHandler for the netmiko webpage) in parallel.

Group of scripts: it should be possible to create a group a script in order to run / schedule batchs of scripts.

Timeline of scripts: let the user create "workflow" of scripts, i.e schedule a script / a group of scripts to run in a predefined order (first script A, then group of scripts B, then script C, etc)

Depending on the browser used and the dimension of the screen, the bootstrap panels are not necessarily aligned. They should have a height of 100% instead of a fixed height in px.

Same issue with the buttons of the "manage devices" webpage, because of the

Migrate database to PostgreSQL

let the user upload a profile pic, save the path + name in the database, and display in the sidebar and wherever necessary

Let the user choose the type of script:

• normal: no yaml file
• j2 template: a yaml file used for all target devices
• Per-node template: the yaml file should have, as first-level keys, the name of the target devices
  R1:
  values R1...
  R2:
  values R2...
  ...
  Rn:
  values Rn...

For each hostname, the jinja2 template uses the specific values stored in the YAML file.

Make eNMS able to act as a syslog server:

• receive the logs and store them in the database
• advanced regex-based filtering of the logs (filter per device, filter per keyword, a keyword can be a string or a regex)
• parse the logs to retrieve the time and give the user a way to look at the log from start time til end time (bootstrap)

When load_merge_candidate or load_replace_candidate is executed on some devices, automatically execute device.compare_config() for all selected devices, as it is the next logical step before commiting.
Display the output of the comparison for each device in the output panel in the middle.

Issue

Received two ModuleNotFoundErrors after installation:

ModuleNotFoundError: No module named 'flask_wtf'

ModuleNotFoundError: No module named 'napalm_base'

Resolution

• Installed flask-wtf integration:

pip install flask-wtf

• Renamed napalm_base to napalm on line 1 in models.py

Recommended Solution

• Add flask-wtf to requirements.txt
• Rename napalm_base to napalm in models.py

Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 2309, in call
return self.wsgi_app(environ, start_response)
File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 2295, in wsgi_app
response = self.handle_exception(e)
File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1741, in handle_exception
reraise(exc_type, exc_value, tb)
File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 2292, in wsgi_app
response = self.full_dispatch_request()
File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1815, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1718, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1813, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1799, in dispatch_request
return self.view_functionsrule.endpoint
File "/var/www/html/project/eNMS/source/admin/routes.py", line 63, in login
if user and cisco_type7.verify(password, user.password):
File "/usr/local/lib/python2.7/dist-packages/passlib/utils/handlers.py", line 757, in verify
self = cls.from_string(hash, **context)
File "/usr/local/lib/python2.7/dist-packages/passlib/handlers/cisco.py", line 366, in from_string
salt = int(hash[:2]) # may throw ValueError
ValueError: invalid literal for int() with base 10: 'am'

Let the user configure GIT parameters for automatically pushing to gitlab/github the results of the script.
=> compare scripts results like configs, pings, traceroutes directly on git.

• if one end of a link is not displayed because of the node filter: deactivate the link
• if there is a match, choose between highlight or undisplay rest
• choose whether to display a label or not
• choose whether the label is display with javascript leaflet mode permanent: true or permanent: false

• When a task has been executed and is not in the database, it is no longer in the database. The edit button should disappear in the task management page, and the state of the task set to Completed.
• Clicking on a task in the calendar should display different things depending on whether it is still active or not. If it is still active, it should display the edit modal. If it is completed, the logs.
• The logs should be displayed in the same way as the comparison for periodic tasks: with drop-down lists for "versions" and "scripts".
• Rescheduling: when the start time, end date or frequency is changed from the edit form, the task should be rescheduled accordingly (currently these changes are not taken into account).
• Date format bug: when changing the start/end dates from the edit modal, the dates are formatted with slash instead of dash => formatting bug.

At the moment, spinning up an eNMS container:

1. Is a three step process.
2. Has a randomly generated name.
3. Does not automatically restart.

I'll raise a PR that resolves these issues.

• create a form for the admin users to configure TACACS (server property, nb of retry, etc)
• in sidebar, user section should be available only to privileged user

There should be a list of default scripts to be activated by the user

• Ping script:
  Ping a subset of devices every x seconds. Email with list of all non-pingable devices.

• Traceroute script:
  Traceroute to a subset of devices every x seconds. Save the result in the database.
  Possibility to compare traceroutes at different period to see the change.

• Config script
  Retrieve the configuration of a subset of devices every x seconds.
  Possibility to compare configs at different period to see the change.

Hello,
I've the following message when I delete a task:
Not Found
The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.

Ragards

Resize map - on 2k screen the map only utilizes half of the screen. It would be nice for it to automatically fill the screen (good for on-going display purposes).

password encryption broke the netmiko / napalm connection to a device.
=> upon sending a script, eNMS should ask the user to provide the credentials (they will not be stored anywhere, we only store a hash of a the password in the database)

Implement a duplication system everywhere where it makes sense:

• duplication of nodes and links
• duplicated of scripts
• duplication of workflows

When the user clicks on Duplicate, make a bootstrap modal appear with a unique Name field for the name of the duplicate object.

Hi,

Could you implement a secure password hashing? I'm not really comfortable with the fact that passwords are visible in plain text in a sqlite file-based database.

As of them, when a task has been created (send a script in a future time or retrieve the getters at periodic interval), there is no way for the user to see which tasks are scheduled, pause them, resume them, or delete them.
All scheduled tasks should be made in the main database and there should be a "Scheduled tasks" webpage where the user can control them.

Login with enable root by default (just with one username and password) would be a great improvement.

This will be nice to have in environments where no secret password is enabled and username/password is granted already with level 15.

• create a napalm-logs webpage to configure and start napalm-logs
• use napalm-logs to retrieve the logs and store them in the SQL database
• create a webpage to display the logs (+ filter with criteria, order, etc) for each device

Do you know about the asynchronous library netdev like netmiko for configuring network devices? It can be really fast if there are many devices to configure.

Though there are prod and debug classes:

https://github.com/afourmy/eNMS/blob/60155577f34022366572373aa0d24d0f0662afc3/source/config.py#L22-L29

Only the latter is used:

https://github.com/afourmy/eNMS/blob/60155577f34022366572373aa0d24d0f0662afc3/source/flask_app.py#L102

I propose an environment variable be used to determine set the type of deployment. e.g prod or debug.

I'm happy to submit a PR for this issue too.

At present the Flask keys for prod and test are set to change-me and key. Furthermore, they're both stored in clear text in the git repo.

https://github.com/afourmy/eNMS/blob/be32c3f75c2bacbd5daa703216d9ca0383ba006d/source/config.py#L22-L29

Keys should instead be retrieved through environment variables and kept out of version control. I'm happy to submit a PR that achieves this.

When netmiko is used to send a script with wrong commands, there is no exception as the ssh connection to the device is working: netmiko logs are the only way to find out what went wrong.
This logs must be saved in a python variable and displayed on the website by redirecting the log stream to a fake sys.stdout StringIO() object.

Something like that:

logger = logging.getLogger("netmiko")
sys.stdout = netmiko_out = StringIO()
ch = logging.StreamHandler(netmiko_out)
ch.setLevel(logging.DEBUG)
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
ch.setFormatter(formatter)
logger.addHandler(ch)

This will let the user know why the script is wrong.

Use simplekml to create a .kmz file to export a project to Google Earth

For each database object, there should be a possibility to edit its properties from the interface:

• Users: to be done
• Scripts: to be done
• Tasks: to be done
• Objects: currently, an object can be edited by filling the object creation form with the name of the object. Downside: the property fields are not pre-populated with the value of the property for the object. There should be a dedicated editing system with a drop-down list of all objects + all properties initiliazed to their current value upon selecting an object of the list with jQuery.

• add physical ports / interfaces to the database
• port creation upon link creation
• port property
• display port in the geographical / logical view, port highlight, port selection from the view, port labels

• Create a Dockerfile to setup eNMS with a PostgreSQL database
• update docs with how to setup PostgreSQL database

create user profile page to

• change name, password + other parameters
• update the profile picture

Some functions in napalm have arguments: ping, traceroute, get_route_to, etc.
They are not currently supported but there should be a specific webpage for these functions, where the user can set values for those arguments in the GUI.

Use FullCalendar.js to display all tasks in a calendar.
When clicking on a task, the results of the task should be displayed in a bootstrap modal just like in the Task management page.

Currently, the getters are stored in text file locally, in the /getters subfolder in the folder of the Flask application. Ideally, they should be stored in the database, so that the user can display them from the website.
Additionnaly, once they are in the database, it would be interesting to have a a wikipedia-like concurrent version system to compare the getters in time, i.e compare the differences between two times.