Another issue I noticed is that if configuration-db, statistics-db, or application-server are down, the script will fail ungracefully and will not even reach this section of the AURA script. It calls to question if this check is really valuable and/or if it should be performed first. Just wanted to share my findings here.

Example:

INFO:Executing the script in Normal execution mode
INFO:Generating a JSessionID
INFO:Generating CSRF Token
INFO:****Collecting Preliminary Data

ERROR:the JSON object must be str, not 'NoneType'
Traceback (most recent call last):
File "py3_sure.py", line 1922, in
controllers = json.loads(getRequestpy3(version_tuple, vmanage_lo_ip, jsessionid , 'system/device/controllers', args.vmanage_port, tokenid))
File "/usr/lib/python3.5/json/init.py", line 312, in loads
s.class.name))
TypeError: the JSON object must be str, not 'NoneType'

JIRA ticket: https://jira-eng-sjc4.cisco.com/jira/browse/SDWANAURA-54

INFO:#26:Check:Cluster:Cluster health
ERROR:'system-ip'
Traceback (most recent call last):
File "sure.py", line 4329, in
services_down, check_result, check_analysis, check_action = criticalCheckthirteen(cluster_health_data)
File "sure.py", line 932, in criticalCheckthirteen
(device['configJson'].pop('system-ip'))

Hi,
I have an important customer who is in preparation phase for SW version upgrades and router OS conversions from Viptela OS to IOS-XE. He has an on-prem environment and I recommended him to run the SURE tool so that we can have a better diagnosis of the network, but unfortunately there has been an error.

Here is the script output:

ITFEBSBSC1VM01:~$ python sure.py -u admin
vManage Password:
#########################################################

AURA SDWAN (SURE) - Version 2.0.0

#########################################################

Performing SD-WAN Audit & Upgrade Readiness

#########################################################

ERROR: Error Collecting Preliminary Data.
Please check error details in log file: sdwan_sure/sure_logs_09_02_2023_12_44_00.log.
If needed, please reach out to tool support at: [email protected], with your report and log file.

And here is the log:

INFO:Executing the script in Normal execution mode
INFO:Generating a JSessionID
INFO:Generating CSRF Token
INFO:****Collecting Preliminary Data

ERROR:'version'
Traceback (most recent call last):
File "sure.py", line 2990, in
controllers_info = controllersInfo(controllers)
File "sure.py", line 355, in controllersInfo
controllers_info[count] = [(device['deviceType']),(device['deviceIP']),(device['version']) ,(device['reachability']),(device['globalState']),(device['timeRemainingForExpiration']), (device['state_vedgeList'])]
KeyError: 'version'

reported as direct upgrade from 19.2.4 to 20.6.x. As per below link, it should be a step upgrade to 20.6.x.

cid:2576524502*[email protected]

CHECK10_ERROR.txt

INFO:#11:Check:Controllers:Validate vSmart/vBond CPU count for scale
ERROR:list index out of range
Traceback (most recent call last):
File "sure.py", line 3880, in
total_cpu_count = int(output['data'][0]['total_cpu_count'])
IndexError: list index out of range

INFO:****Collecting Preliminary Data

ERROR:'timeRemainingForExpiration'
Traceback (most recent call last):
File "sure.py", line 2499, in
controllers_info = controllersInfo(controllers)
File "sure.py", line 327, in controllersInfo
controllers_info[(device['host-name'])] = [(device['deviceType']),(device['deviceIP']),(device['version']) ,(device['reachability']),(device['globalState']),(device['timeRemainingForExpiration']), (device['state_vedgeList']) ]
KeyError: 'timeRemainingForExpiration'

During initial data capture, the tools fails with:

ERROR:'version'
Traceback (most recent call last):
File "sure.py", line 9362, in
controllers_info = controllersInfo(controllers)
File "sure.py", line 327, in controllersInfo
controllers_info[(device['host-name'])] = [(device['deviceType']),(device['deviceIP']),(device['version']) ,(device['reachability']),(device['globalState']),(device['timeRemainingForExpiration']), (device['state_vedgeList']) ]
KeyError: 'version'

Trigger is a controller on transition state, without full information, possibly on partial install state
The fix will have two parts:

1. handle properly this condition and gracefully skip over those entries
2. add a check reporting how many entries like this have been found, with their IP addresses, so user can check on them

The ‘json’ library is not available in python2 on 20.5.x vManages and the tool recommends to try python3 instead. If you do this, the ‘Queue’ import fails. This can be fixed by replacing ‘import Queue’ with ‘import queue as Queue’.

It would be good to put the results in a table format, like cpu, memory, interface , control etc.., against recommended and measured value.

Implement check based on output from command in CSCvx69668
Report critical error if DB is >=5GB

INFO:#4:Check:vManage:CPU Count
ERROR:local variable 'check_result' referenced before assignment
Traceback (most recent call last):
File "./sure.py", line 8528, in
check_result, check_analysis, check_action = criticalCheckfour(cpu_count, vedge_count, dpi_status, server_type)
File "./sure.py", line 568, in criticalCheckfour
return check_result, check_analysis, check_action
UnboundLocalError: local variable 'check_result' referenced before assignment

For 20.3, we should see v0.A.9. If the returned store version is older:

2021-08-21 18:22:21.909+0000 INFO [o.n.i.d.DiagnosticsManager] NeoStore v0.A.8

It may point to a previous upgrade failure.

This scenario will cause 20.6 upgrade failures:
org.neo4j.kernel.impl.storemigration.StoreUpgrader$UnexpectedUpgradingStoreVersionException: Not possible to upgrade a store with version 'v0.A.8' to current store version SF4.0.0 (Neo4j 4.1.7).

This needs manual recovery

for tool run, if check is bypassed or not done (could be not applicable), the total count does not show it... so tool output shows "32 checks", but then run results will show 30 passed, 0 failed

The count should match, and we should add a "bypassed" or "not applicable" check count

CJF-AURA-SDWAN-error.txt

Script does auth sucessfully , but fails token generation, with a JSON error
when tested manually it works

INFO:Generating a JSessionID
INFO:Generating CSRF Token
ERROR:No JSON object could be decoded
Traceback (most recent call last):
File "./sure.py", line 2471, in
tokenid = CSRFToken(vmanage_lo_ip,jsessionid,args.vmanage_port)
File "./sure.py", line 224, in CSRFToken
tokenid = json.loads(tokenid)
File "/usr/lib/python2.7/json/init.py", line 339, in loads
return _default_decoder.decode(s)
File "/usr/lib/python2.7/json/decoder.py", line 364, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python2.7/json/decoder.py", line 382, in raw_decode
raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded

The documentation in the GitHub README says to use py2_sure.py for vManages running version 20.5.x, but line 3316 prevents all checks on the 20.5.x versions.

INFO:#5:Check:vManage:ElasticSearch Indices status
ERROR:the JSON object must be str, not 'NoneType'
Traceback (most recent call last):
File "sure.py", line 3696, in
es_indexes_one = json.loads(getRequestpy3(version_tuple,vmanage_lo_ip, jsessionid, 'management/elasticsearch/index/info', args.vmanage_port, tokenid))
File "/usr/lib/python3.5/json/init.py", line 312, in loads
s.class.name))
TypeError: the JSON object must be str, not 'NoneType'

Enhancement request: when the tool is executed in container, for example in vDoctor, add support for dynamic importing of PDF writer library, and generate report in PDF instead of text

I am documenting on TechZone how to handle some of the failures we notice from the AURA script. When trying to recreate some of the failed states I noticed an issue with the services check, criticalCheckseven.

I killed the service "data collection agent" in my lab but kept it enabled to trigger the failed state.

vmanage_20_6_4# request nms all status
NMS service proxy
Enabled: true
Status: running PID:8149 for 944358s
NMS service proxy rate limit
Enabled: true
Status: running PID:9810 for 944358s
NMS application server
Enabled: true
Status: running PID:1530 for 64879s
NMS configuration database
Enabled: true
Status: running PID:26175 for 65364s
NMS coordination server
Enabled: true
Status: running PID:3585 for 65058s
NMS messaging server
Enabled: true
Status: running PID:10765 for 944395s
NMS statistics database
Enabled: true
Status: running PID:24440 for 65440s
NMS data collection agent
Enabled: true
Status: not running <<<<<<<<Induced failure state here.
NMS CloudAgent v2
Enabled: true
Status: running PID:10415 for 944396s
NMS cloud agent
Enabled: true
Status: running PID:7627 for 944354s
NMS SDAVC server
Enabled: false
Status: not running
NMS SDAVC proxy
Enabled: true
Status: running PID:7630 for 944354s

But the AURA report shows that this check is SUCCESSFUL.

            INFO:#07:Check:vManage:Validate all services are up
            INFO:#07: Check result:   SUCCESSFUL                                                                                                    <<<<<<False SUCCESS
            INFO:#07: Check Analysis: All enabled services are running
            INFO:#07: Status of all the services:
            NMS service proxy
                                            Enabled: true
                                            Status: running PID:8149 for 879846s
            NMS service proxy rate limit
                                            Enabled: true
                                            Status: running PID:9810 for 879846s
            NMS application server
                                            Enabled: true
                                            Status:  running PID:1530 for 368s
            NMS configuration database
                                            Enabled: true
                                            Status: running PID:26175 for 852s
            NMS coordination server
                                            Enabled: true
                                            Status: running PID:3585 for 546s
            NMS messaging server
                                            Enabled: true
                                            Status: running PID:10765 for 879883s
            NMS statistics database
                                            Enabled: true
                                            Status: running PID:24440 for 928s
            NMS data collection agent
                                            Enabled: true
                                            Status:  not running                                                                                         <<<<<<<<<<<<<<<<<<Should be considered failed for this check
            NMS CloudAgent v2
                                            Enabled: true
                                            Status: running PID:10415 for 879884s
            NMS cloud agent
                                            Enabled: true
                                            Status:  running PID:7627 for 879842s
            NMS SDAVC server
                                            Enabled: false
                                            Status: not running
            NMS SDAVC proxy
                                            Enabled: true
                                            Status:  running PID:7630 for 879842s

I put this in my lab and identified the issue is this. In the function criticalCheckseven we are overwriting the check_result = 'Failed' on the next successful check. Although we build a list of failed services we don't use that.

            def criticalCheckseven(nms_status1):
nms_status = nms_status1.split('NMS')
nms_failed = []
for nms in nms_status:
    print(nms)
    if 'true' in nms and 'not running' in nms:
        nms_failed.append(nms.split('\t')[0].strip())
        check_result = 'Failed'
        check_analysis = 'Enabled service/s not running'
        check_action = 'It is advisable to investigate why a service is being reported as failed. Please  restart the process or contact TAC for further help'
    else:
        check_result = 'SUCCESSFUL'
        check_analysis = 'All enabled services are running'
        check_action = None
return nms_status1, nms_failed, check_result, check_analysis, check_action

When I pulled this function to test in my lab, I could see the following results. We have the correct service failed in the list of nms_failed but since we overwrite the check_result with SUCCESSFUL on the next service check, we report SUCCESSFUL on the script.

            nms_failed - ['data collection agent']
            check_result - SUCCESSFUL
            check_analysis - All enabled services are running
            check_action - None

In the below function where we call criticalCheckseven, we only use check_result to determine success or failure. It may be better to check the length of our nms_failed list.

            #Check:vManage:Validate all services are up
            check_count += 1
            check_count_zfill = zfill_converter(check_count)
            if args.quiet == False and args.debug == False and args.verbose == False:
                                            print(' Critical Check:#{}'.format(check_count_zfill))
            if args.debug == True or args.verbose == True:
                            print(' #{}:Checking:vManage:Validate all services are up'.format(check_count_zfill))
            check_name = '#{}:Check:vManage:Validate all services are up'.format(check_count_zfill)
            pre_check(log_file_logger, check_name)
            try:
                            nms_data, nms_failed, check_result, check_analysis, check_action =   criticalCheckseven()
                            if check_result == 'Failed':
                                            critical_checks[check_name] = [check_analysis, check_action]
                                            check_error_logger(log_file_logger, check_result, check_analysis, check_count_zfill)
                                            log_file_logger.error('#{}: List of services that are enabled but not running:\n{}'.format(check_count_zfill, nms_failed))
                                            log_file_logger.error('#{}: Status of all services  :\n{}\n'.format(check_count_zfill, nms_data))
                                            report_data.append([str(check_count),check_name.split(':')[-1],check_result,check_analysis,str(check_action)])
                                            if args.debug == True:
                                                            print('\033[1;31m ERROR: {} \033[0;0m \n\n'.format(check_analysis))

                            else:
                                            check_info_logger(log_file_logger, check_result, check_analysis, check_count_zfill)
                                            log_file_logger.info('#{}: Status of all the services:\n{}\n'.format(check_count_zfill, nms_data))
                                            report_data.append([str(check_count),check_name.split(':')[-1],check_result,check_analysis,str(check_action)])
                                            if args.debug == True:
                                                            print(' INFO:{}\n\n'.format(check_analysis))
                            json_final_result['json_data_pdf']['description']['vManage'].append({'analysis type': '{}'.format(check_name.split(':')[-1]),
                                                                                                                                                                                                                            'log type': '{}'.format(result_log['Critical'][check_result]),
                                                                                                                                                                                                                            'result': '{}'.format(check_analysis),
                                                                                                                                                                                                                            'action': '{}'.format(check_action),
                                                                                                                                                                                                                            'status': '{}'.format(check_result),
                                                                                                                                                                                                                            'document': ''})
            except Exception as e:
                            print('\033[1;31m ERROR: Error performing {}. \n Please check error details in log file: {}.\n If needed, please reach out to tool support at: [[email protected]](mailto:[email protected]), with your report and log file. \033[0;0m  \n\n'.format(check_name, log_file_path))
                            log_file_logger.exception('{}\n'.format(e))

Could we add as well an another way to copy the file?

Eg picking us

wget https://raw.githubusercontent.com/CiscoDevNet/sure/main/python3/py3_sure.py

Or

wget https://raw.githubusercontent.com/CiscoDevNet/sure/main/python2/py2_sure.py

in neo4j we need to check the configdb size , and put a warning incase slicing of DB is required.

Following up with Manikandan to get more clarity on this issue.

INFO:#30:Check:Cluster:Intercluster communication
ERROR:'function' object has no attribute 'isAlive'
Traceback (most recent call last):
File "sure.py", line 4436, in
if criticalCheckseventeenpy3.isAlive():
AttributeError: 'function' object has no attribute 'isAlive'
INFO:Logging out of the Session
INFO:Successfully closed the connection

Problem Description:
• customer is planning to upgrade the infra to version 20.9.x
• as recommended by the BCS team ( Cisco Business critical service team ) , customer requested to execute "Aura Script”.
• Aura script identified the "CPU Clock configured is 2.6 instead of 2.8"
• CloudOps upgraded the infra for customer from 16 CPU's to 32 CPU's , but clock remain the same.
• customer want to confirm if they can move forward with CPU clock 2.6 or otherwise CloudOps required to upgrade the infra to VM type with clock 2.8.

DSV hosted on azure
that makes sense usually we can find out type of hypervisor its hosted in kernl logs

In Kern.log while bootup it will show Microsoft
or get a linux command to identify if Azure infrastructure. (Get source command)

ERROR:the JSON object must be str, not 'NoneType'
Traceback (most recent call last):
File "sure.py", line 3821, in
es_indices_est = json.loads(getRequestpy3(version_tuple,vmanage_lo_ip, jsessionid, 'management/elasticsearch/index/size/estimate', args.vmanage_port, tokenid))
File "/usr/lib/python3.5/json/init.py", line 312, in loads
s.class.name))

might have to update the script because it is not using the data sent by

def criticalChecktwentyone(version):
success, analysis, action = validateServerConfigsFile()
if not success:
check_result = 'Failed'
check_analysis = 'Failed to validate the server_configs.json.'
check_action = '{}'.format(analysis)
else:
check_result = 'SUCCESSFUL'
check_analysis = 'Validated the server_configs.json.'
check_action = None
log_file_logger.info('Validated the server_configs.json.')

return check_result, check_analysis, check_action

mt-vmanage1:~/sdwan_sure$ cat sure_report_22_01_2022_08_15_31.txt
Cisco SDWAN AURA v1.0.8 Report

Cisco SDWAN AURA command line tool performed a total of 32 checks at different levels of the SDWAN overlay.

Reach out to [email protected] if you have any questions or feedback

Summary of the Results:

mt-vmanage1:~/sdwan_sure$ cat sure_logs_22_01_2022_08_15_31.log
INFO:Executing the script in Normal execution mode
INFO:Generating a JSessionID
INFO:Generating CSRF Token
INFO:****Collecting Preliminary Data

ERROR:'state_vedgeList'
Traceback (most recent call last):
File "./sure.py", line 2499, in
controllers_info = controllersInfo(controllers)
File "./sure.py", line 327, in controllersInfo
controllers_info[(device['host-name'])] = [(device['deviceType']),(device['deviceIP']),(device['version']) ,(device['reachability']),(device['globalState']),(device['timeRemainingForExpiration']), (device['state_vedgeList']) ]
KeyError: 'state_vedgeList'
mt-vmanage1:~/sdwan_sure$

To allow GUI integration with vDoctor, add suport for JSON export format

Tool is generating alert if WARN or ERROR messages are detected. This is generating unnecessary alerts, and WARN level should be ignored
Also, check is not validating how old is the message. it should ignore anything older than 14 days

In order to shorten the length of the script, the script needs to be split into two versions Python2 and Python3.

vedge count is never initialized as json parsing fails on null returned by API call

Hi all,

I was struggling a little with getting this script to work. I had the consistently the same error in the logs:

INFO:Executing the script in Verbose execution mode
INFO:Generating a JSessionID
ERROR:list index out of range
Traceback (most recent call last):
  File "sure.py", line 8354, in <module>
    jsessionid = generateSessionID(vmanage_lo_ip, args.username, password, args.vmanage_port)
  File "sure.py", line 211, in generateSessionID
    jsessionid = (login[3].split('=')[1][0:-1])
IndexError: list index out of range

The problem turns out to be in the password, because the user input was not sanitized. I had several special characters in the password, including a quote sign etc.
Perform error handling would've dealt with this issue in a better way. More importantly, sanitizing user input would prevent the script from breaking in the first place.

holistic view if there is any major error found by the script or not.

Check Results:
Total Checks Passed: 26
Total Checks Failed: 4

The installation instructions mention installation via Git but Git is not available via vShell. The installation instruction should also mention what directory to install to.

We should monitor the memory and cpu per nms service as well like how we do for the neo4j, also wildfly, statsdb

INFO:#22:Check:vManage:Disk controller type
ERROR:local variable 'check_result' referenced before assignment
Traceback (most recent call last):
File "sure.py", line 4208, in
check_result, check_analysis, check_action = infoCheckone(server_type, disk_controller)
File "sure.py", line 1387, in infoCheckone
return check_result, check_analysis, check_action
UnboundLocalError: local variable 'check_result' referenced before assignment

Reduces the lines of code from 15k to 6k.

Tool may report JSESSION error, when password contains the sequence $$ or a +
Logs show:

Traceback (most recent call last):
File "sure.py", line 2460, in <module>
jsessionid = generateSessionID(vmanage_lo_ip, args.username, password, args.vmanage_port)
File "sure.py", line 210, in generateSessionID
jsessionid = (login[3].split('=')[1][0:-1])
IndexError: list index out of range

Intercluster communication we may need to tell the Roundtrip delay