stdevel / satprep Goto Github PK

Python toolkit for automating system maintenance and generating CSV/PDF patch reports along with Spacewalk, Red Hat Satellite 5.x or SUSE Manager

License: GNU General Public License v3.0

TeX 2.56% Python 97.44%

spacewalk suse-manager maintenance maintenance-tasks maintenance-reports iso27001 linux redhat centos satellite

satprep's Introduction

satprep

satprep is a Python toolkit for automating system maintenance and generating CSV/PDF patch reports for systems managed with Spacewalk, Red Hat Satellite 5.x or SUSE Manager.

This can be very useful if you need to document software changes due to IT certifications like ISO/IEC 27001:2005 or many other.

After doing maintenance tasks this toolkit can create detailed change reports per host. Before starting maintenance you can also automate scheduling downtime for your systems and creating VM snapshots.

I'm currently re-developing this toolkit for Foreman/Katello and Red Hat Satellite 6.x - the project is called katprep. So, I'm afraid I won't spent too much time on this toolkit, anymore..

Supported software

The following monitoring suites are supported:

Nagios
Icinga
Thruk
Shinken
of course this also works in combination with the Swiss Army knife OMD.

As this toolkit uses the Python bindings of libvirt a wide range of Hypervisors can be used - including:

KVM/Qemu
Xen
VMware vSphere ESXi
Microsoft Hyper-V
...and many more

What does a report look like?

A maintenance report looks like this:

How does it work?

The toolkit consists of four scripts:

satprep_install_custominfo.py - installs necessary custom information (see above) for your hosts managed with Spacewalk, Red Hat Satellite or SUSE Manager. You will need to execute this script once to make sure that all information can be assigned
satprep_snapshot.py - creates an inventory of the current available patches and errata to your system landscape. It gathers the following information:

system hostname and IP
system virtualization guest status
system owner / cluster member / monitoring / backup / anti-virus status (optional)
errata information including name, date, description, advisory type (security/bugfix/enhancement update) and even whether a reboot is required
also regular patch information (optional)

satprep_prepare_maintenance.py - schedules monitoring downtimes and creates VM snapshots, also makes sure that all required preparations are done before you begin with your work
satprep_diff.py - creates the delta, required to create the maintenance reports

Make sure to follow this procedure to document your maintenance tasks:

do a complete patch/errata inventory of your landscape: ./satprep_snapshot.py
notice that a CSV report was created: errata-snapshot-report-$RHNhostname-YYYYMMDD-HHMM.csv
prepare maintenance; automatically create VMware snapshots and monitoring downtimes and verify them: ./satprep_prepare_maintenance.py snapshot.csv (optional)
complete your system maintenance tasks (patch and reboot systems, etc.)
do another complete patch/errata inventory: ./satprep_snapshot.py
create a difference report and host reports: ./satprep_diff.py *.csv

Afterwards the reports are stored in /tmp.

For gathering optional semantic information (e.g. backup and monitoring) the script makes usage of the custom system information feature of Spacewalk, Red Hat Satellite or SUSE Manager. After installing the custom keys using the satprep_install_custominfo.py utility you can assign the following information per host (only a selection):

SYSTEM_OWNER - server responsible in your team
SYSTEM_MONITORING - monitoring state (0 or empty = disabled, 1 = enabled)
SYSTEM_BACKUP - defines whether the host is protected using backups (0 or empty = no, 1 = yes)
SYSTEM_ANTIVIR - defines whether the host is protected against viruses (0 or empty = no, 1 = yes)

See the wiki for more details about the particular scripts.

Requirements

satprep needs Python 2.6 or newer - it runs on EL5/6 machines without adding additional software repositories. The following additional Python modules are used:

libvirt
xmlrpclic (shipped with rhnlib)
requests
unidecode

The toolkit needs the pdflatex binary which is usually part of the LaTeX or TeX Live software set provided by your distributor.

Check out the wiki for more detail information.

Installation and usage

See the wiki for more details about the particular scripts.

Example workflow

Create an inventory for all managed hosts, including errata and regular patch information:

$ ./satprep_snapshot.py -p

Prepare maintenance for affected hosts (optional):

$ ./satprep_prepare_maintenance.py errata-snapshot*.csv
...
$ ./satprep_prepare_maintenance.py -V errata-snapshot*.csv

Patch your systems, reboot them, verify functionality, etc.:

Create another snapshot afterwards:

$ ./satprep_snapshot.py -p

Create maintenance reports with the information from two snapshot reports:

$ ./satprep_diff.py -x errata-diff-report*.csv

Or create the same reports with different page orientation, an custom logo (e.g. company logo) and a custom footer:

$ ./satprep_diff.py -x errata-diff-report* -p potrait -i /opt/tools/myCompany.jpg -f "myCompany maintenance report"

satprep's People

Contributors

Stargazers

Watchers

Forkers

okin borland667 pankajps platipusica itesgit pradeepbhuyan

satprep's Issues

RHEL kernel updates aren't marked to require reboot

Unfortunately Red Hat isn't marking Kernel errata to require reboots like CEFS does for CentOS. As a result reports are telling that no reboot needs to be executed - which is wrong.

Dedicated monitoring and virtualization host definitions not included in snapshot report

Currently dedicated monitoring and virtualization host definitions (custominfos SYSTEM_VIRT_HOST, SYSTEM_VIRT_HOST_AUTH, SYSTEM_MONITORING_HOST, SYSTEM_MONITORING_HOST_AUTH) are not included in snapshot reports.
As a result it is sufficient to enter those information manually.

Add parameter for preparing maintenance for prod or non-prod systems only

Currently satprep_prepare_maintenance.py prepares maintenance for all system types. As there is a custom key SYSTEM_PROD it would be great have parameters for only preparing prod or non-prod systems (e.g. patching non-prod before patching prod systems).

The idea is to implement parameters -p / --prod-only and -D / --nonprod-only which control this behavior. The script needs to be able to read the report column system_prod for this.

Diff reports include new patches as installed patches

Diff reports currently include new patches as installed patches even if they are not installed.

Steps to reproduce:
1.Sync repo
2.Create snapshot
3.Patch systems
4.Sync repo including new patches
5.Create another snapshot and diff report

satprep_snapshot.py aborts with non-iterable empty object

satprep_snapshot.py with an non-iterable empty object

Monitoring host/host auth information not included in snapshot report

Currently monitoring host/host auth information are not included in snapshot reports.

Temporarily disable DRS / Storage DRS for satprep_wa_vcvms.py

The script satprep_wa_vcvms.py crashes if a VM was moved to another host or storage once it started:

INFO:satprep_wa_vcvms:Checked 270 of 527 VMs so far...
Traceback (most recent call last):
  File "./satprep_wa_vcvms.py", line 251, in <module>
    main(options)
  File "./satprep_wa_vcvms.py", line 95, in main
    thisVM = myVC.get_vm_by_path(vm)
  File "/usr/lib/python2.6/site-packages/pysphere-0.1.8-py2.6.egg/pysphere/vi_server.py", line 291, in get_vm_by_path
    FaultTypes.OBJECT_NOT_FOUND)
pysphere.resources.vi_exception.VIException: [Object Not Found]: Could not find a VM with path '[Prod 01] MYSRV/MYSRV.vmx'

It would be great if the script checks the used clusters and storage clusters first and temporarily disables automatic load-balancing.

Add script for automatic Nagios/Icinga/Thruk/Shinken downtime scheduling

It would be great to have a script satprep_schedule_downtime.py which automatically schedules downtimes in Nagios/Icinga/Thruk/Shinken for hosts needing reboots.

Automatically create verification log during satprep_prepare_maintenance

Currently a verify log is created when executing satprep_prepare_maintenace.py -V - it would be great if this is also done automatically.

Simulation / dry-run not working for satprep_install_custominfos.py

Currently -n / --dry-run isn't working for satprep_install_custominfos.py in combination with -u / --uninstall:

$ ./satprep_install_custominfos.py -un
$

Script for prepare maintenance

As satprep also has information about virtual machines it would be great if it also automatically creates VM snapshots.
The idea is to implement a script satprep_prepare_maintenance.py that combines creating snapshots and scheduling downtimes.
It would also be awesome to have an option to unschedule downtimes and removing snapshots.

Implement UTF-8 support

satprep_snapshot.py is currently not able to handle umlauts:

UnicodeEncodeError: 'ascii' codec can't encode character u'\xfc' in position 1: ordinal not in range(128)

satprep not supported on Spacewalk 2.4 or newer

Currently satprep is not supported for Spacewalk 2.4 or newer:

satprep_shared.APILevelNotSupportedException: Your API version (16) does not support the required calls. You'll need API version 1.8 (11.1) or higher!

It is sufficient to test and alter the list of supported API levels.

dry-run is ignored for deleting huge amount of actions

The --dry-run pararemeter is ignored if a huge amount of actions is being removed (workaround).

satprep_prepare_maintenance: add comma separation for -e / --exclude

Currently you need to supply multiple -e / --exclude parameters when exluding systems during maintenance preparation with satprep_prepare_maintenance.py.

It would be nice to have a comma separation to make commands shorter - so:
./satprep_prepare_maintenance.py -e a -e b -e c -e d
could also be written like this:
./satprep_prepare_maintenance.py -e a,b,c,d

Not an issue, more like a feature request

I'm not sure if I'm putting this in the right place, hopefully I am.
Would it be possible to add to the pdf/diff that gets created, the CVE # that the installed patch references? That's one of the requirements of my security report.
If this is the wrong way to go about asking for this, please advise and I'll post correctly.
Thanks very much!

Long table breaks LaTeX layout

Currently having a long patch/erratum table breaks the LaTeX layout - especially the footer.

This seems to be a minor LaTeX template issue.

satprep_snapshot.py - default columns includes "errata_date" twice

By default the column "errata_date" is included twice.

Script for doing complex maintenance

As satprep already offers the possibility to (un-)schedule downtimes and create/remove VM snapshots it would be great if it does more complex maintenance in an automated way. I think about nasty things like:

patch applications
restart particular daemons
execute third-party scripts
force rechecking monitoring
validating that required services are "OK" in Nagios, Icinga and Shinken

Custom info keys are not escaped

Custom info keys are not escaped, resulting in aborts sometimes. E.g. if a newline is part of a custom info key, an error is created:

...
Traceback (most recent call last):
  File "./satprep_snapshot.py", line 540, in <module>
    main(options)
  File "./satprep_snapshot.py", line 127, in main
    process_system(client, key, writer, system)
  File "./satprep_snapshot.py", line 151, in process_system
    process_errata(client, key, writer, system)
  File "./satprep_snapshot.py", line 358, in process_errata
    writer.writerow(valueSet)
_csv.Error: need to escape, but no escapechar set

Debugging is kinda hard and real a pain if you have hundreds of hosts..

Use logger functionality

It would be better to use a logger instead of manual printing messages if debug mode is enabled

RPM spec file

It would be handy to have a RPM package for easily deploying satprep.

Idea: relocate CSV intelligence in latex/csvtools

Another (much nicer) possibility to create host patch reports is to use latex/csvtools instead of the current implementation (in which satprep_diff.py does all the stuff).

satprep_prepare_maintenance.py is not working

./satprep_prepare_maintenance.py -k -a auth -g Test_for_satprep -K errata-snapshot-report-localhost-20151217-1433.csv

Haha, you're funny.

Can you have a look please

Anti-virus and backup boxes and notes aren't integrated in PDF reports

Anti-virus and backup states and notes defined by host custom infos aren't integrated in PDF reports.

satprep_snapshot.py - inventory of large system landscapes aborts: "503 Service Temporarily Unavailable"

If an inventory of a large system landscape is created the script aborts after at about 10 minutes with the following error:

Traceback (most recent call last):
  File "./satprep_snapshot.py", line 254, in <module>
    updates = client.system.listLatestUpgradablePackages(key, system["id"])
  File "/usr/lib64/python2.6/xmlrpclib.py", line 1199, in __call__
    return self.__send(self.__name, args)
  File "/usr/lib64/python2.6/xmlrpclib.py", line 1489, in __request
    verbose=self.__verbose
  File "/usr/lib64/python2.6/xmlrpclib.py", line 1243, in request
    headers
xmlrpclib.ProtocolError: <ProtocolError for localhost/rpc/api: 503 Service Temporarily Unavailable>

It seems like there is a timeout which can't be changed for XMLRPC API calls: https://access.redhat.com/solutions/696783
The recommendation from Red Hat is to split large calls in bunches - so the idea is to implement a re-login after ~5-10 hosts when satprep_snapshot.py discovers large environments.

Optimize code structure

Currently the whole code is implemented in main() - it would be better to create functions to optimize readability.

Problem w/ satprep_install_custominfos.py

I get "No handlers could be found for logger "satprep". I'm running python 2.6.6, and would appreciate any help you could supply.

Idea: Implement uninstall function in satprep_install_custominfos.py

It's a good idea to implement a uninstall function in satprep_install_custominfos.py. Using one command it would be possible to completely remove all custom infos used by satprep.

satprep_patch_freeze.py crashed when profile name != hostname

Currently, satprep_patch_freeze.py crashes when a system's profile name differs from the hostname - e.g.:

# ./satprep_patch_freeze.py -g SAP -l sap -n -a sat.auth
ERROR:satprep_patch_freeze:Unable to scan system 'mysapsrv01', check hostname and profile name!

Executable bit missing for some scripts

When cloning satprep from ZIP not all Python scripts have the executable bit. This needs to be fixed.

Add option for scheduling downtimes for complete hostgroups

When preparing downtimes for large system landscapes it might be easier to schedule downtimes for complete hostgroups instead on per host basis. It would be great to add an option to satprep_prepare_maintenance.py to address this.

satprep_wa_vcvms.py is dropping vms that are managed by spacewalk

After creating auth files for spacewalk and vcenter, I ran ./satprep_wa_vcvms.py -A vc -S provis1.domain.com -d -a ~/mon.auth -s pcispacewalk.domain.com. The script finds all the VMware hosts, and finds all the linux guests, but it drops all of the linux guests that are currently managed by spacewalk. The names are the same in spacewalk as in vcenter, so I'm not sure why they're being dropped.

DEBUG:satprep_wa_vcvms:Args: ['./satprep_wa_vcvms.py']
DEBUG:satprep-shared:Using authfile
DEBUG:satprep-shared:File permission matches 0600
DEBUG:satprep-shared:Using authfile
DEBUG:satprep-shared:File permission matches 0600
send: 'POST /rpc/api HTTP/1.0\r\nHost: pcispacewalk.domain.com\r\nUser-Agent: xmlrpclib.py/1.0.1 (by www.pythonware.com)\r\nContent-Type: text/xml\r\nContent-Length: 219\r\n\r\n'
send: "<?xml version='1.0'?>\n<methodCall>\n<methodName>auth.login</methodName>\n<params>\n<param>\n<value><string>satprep</string></value>\n</param>\n<param>\n<value><string>Sk1ttle$</string></value>\n</param>\n</params>\n</methodCall>\n"
reply: 'HTTP/1.1 200 OK\r\n'

DEBUG:satprep_wa_vcvms:'msprodweb3' dropped as it is not managed by Satellite
DEBUG:satprep_wa_vcvms:Current array for host 'jaxucsesx3.domain.com': []
DEBUG:satprep_wa_vcvms:'msprodweb2' dropped as it is not managed by Satellite
DEBUG:satprep_wa_vcvms:Current array for host 'jaxproucsesx4.domain.com': []
DEBUG:satprep_wa_vcvms:'msprodact2' dropped as it is not managed by Satellite
DEBUG:satprep_wa_vcvms:Current array for host 'jaxucsesx2.domain.com': []
DEBUG:satprep_wa_vcvms:'msprodact1' dropped as it is not managed by Satellite
DEBUG:satprep_wa_vcvms:Current array for host 'jaxucsesx3.domain.com': []
DEBUG:satprep_wa_vcvms:'msprodldap' dropped as it is not managed by Satellite
DEBUG:satprep_wa_vcvms:Current array for host 'jaxucsesx3.domain.com': []
DEBUG:satprep_wa_vcvms:'msprodrabbit' dropped as it is not managed by Satellite
DEBUG:satprep_wa_vcvms:Current array for host 'jaxucsesx1.domain.com': []

Thanks

the diff run fails with a missing file

When I run the satprep_diff.py I see:

satprep_diff.py errata-snapshot-report-xxxxxxxxxxx-20150802-*csv
INFO:satprep_diff:Assuming file1 ('errata-snapshot-report-xxxxxxxxx-20150802-0059.csv') is the first snapshot.
INFO:satprep_diff:Snapshot and monitoring checkboxes won't be pre-selected as we don't have a valid .vlog!
Traceback (most recent call last):
  File "./satprep_diff.py", line 492, in <module>
    main(options)
  File "./satprep_diff.py", line 196, in main
    f_log = open(options.verificationLog, 'r')
IOError: [Errno 2] No such file or directory: ''

I also attempted using -x and both of the csv files, same result.

Running under RHEL 6.6 x86_64 Linux.

Create snapshots before scheduling downtimes / increase default downtime timeframe

Currently satprep_prepare_maintenance.py schedules downtimes before creating snapshots. When preparing maintenance for large system landscapes this means that the default downtime timeframe of 2 hours might already be exceeded after creating snapshots (because libvirt isn't able to create snapshots asynchronous).

So changing the order and increasing the downtime timeframe should do the trick.

Idea: integrate scheduling downtime in ``satprep_snapshot.py``?

It would be nice to integrate the scheduling downtime feature into satprep_snapshot.py - currently scheduling downtime is done using satprep_schedule_downtime.py. As a result the downtime is scheduled but not mentioned in the snapshot report - which means that the check list isn't pre-selected.

Enhance exclude functionality for satprep_prepare_maintenance

Currently the -e / --exclude parameters need the full system name specified in Satellite. It would be great if also wildcards and case-insensitive names are supported.

Freezing RHN channels fails

When freezing RHN channels, satprep_patch_freeze.py fails with the following error:

INFO:satprep_patch_freeze:Cloning child-channel 'rhn-tools-rhel-x86_64-server-6' as 'rhn-tools-rhel-x86_64-server-6.sp-sap-2015-11-16'
ERROR:satprep_patch_freeze:Unable to clone base-channel: redstone.xmlrpc.XmlRpcFault: Channels names must not start with 'rhn'

The idea is to prepend the prefix instead of appending it.

satprep_patch_freeze.py not working for systems without base channel

$ ./satprep_patch_freeze.py -a auth -g Test_for_Patching
ERROR:satprep_patch_freeze:Unable to scan system 'dev-stmnt-app-02', check hostname and profile name!
ERROR:satprep_patch_freeze:Unable to scan system 'dev-stmnt-db-01', check hostname and profile name!
ERROR:satprep_patch_freeze:Unable to scan system 'dev-stmnt-idx-01', check hostname and profile name!
ERROR:satprep_patch_freeze:Unable to scan system 'dev-stmnt-app-03', check hostname and profile name!
ERROR:satprep_patch_freeze:Unable to scan system 'dev-stmnt-proxy-01', check hostname and profile name!
ERROR:satprep_patch_freeze:Unable to scan system 'dev-stmnt-db-02', check hostname and profile name!
ERROR:satprep_patch_freeze:Unable to scan system 'dev-stmnt-app-01', check hostname and profile name!
Traceback (most recent call last):
  File "./satprep_patch_freeze.py", line 313, in <module>
    main(options)
  File "./satprep_patch_freeze.py", line 237, in main
    remapSystems(client, key)
  File "./satprep_patch_freeze.py", line 165, in remapSystems
    else: myNewBase = options.targetLabel+"-"+options.targetDate+"."+myBase["label"]
KeyError: 'label'

Failure doing snapshot creation stops satprep_prepare_maintenance script

Currently a failure while creating snapshots stops the script. It would be better if only an error message is printed.

VMware clusters are ignored in satprep_wa_vcvms.py

Currently satprep_wa_vcvms.py does not include cluster information which results in incorrect libvirt URIs - e.g.:

vpx://myvc/mydc/myesx

instead of:

vpx://myvc/mydc/mycluster/myesx

Implement unschedule function for satprep_schedule_downtime.py

It would nice to also unschedule hosts (e.g. in case of prematurly finished maintenance tasks)

Getting clone error while cloning

Getting following issue while cloning. Can you please fix this.

ERROR:satprep_patch_freeze:Unable to clone base-channel: redstone.xmlrpc.XmlRpcFault: unhandled internal exception: The channel parameter rhn-tools-rhel-x86_64-server-6 clone from 2015-12-03 is invalid. It must be at least 6 characters long, begin with a letter, and contain only lowercase letters, digits, '-', ' / ', '_' and '.'. Also, it cannot begin with 'rhn', 'redhat', or 'red hat'.

Exclude locked systems

Currently satprep_snapshot.py includes all systems in snapshot reports. System can also be locked in Spacewalk to avoid automatic maintenance - it would be great to exclude locked systems.

Exclude systems and groups in satprep_snapshot.py

It would be great to have the possibility to exclude systems and groups in satprep_snapshot.py. Parameter ideas:

-e / --exclude
-g / --group
-E / --exclude-group
-H / --include-host

Include regular patch information per default in satprep_snapshot.py

Currently satprep_snapshot.py only includes errata information. Using -p it will also include information about conventional patches. Anyhow, this is mostly used so it also could be considered as default flag.

Patch freezing not possible for RHN Satellite 5.7

Cloning channels with satprep_patch_freezy.py is not working on RHN Satellite 5.7. It seems like the automatic generated channel names are too long:

$ tail /var/log/rhn/rhn_web_api.log
...
[2015-07-10 10:23:03,944] ERROR - REQUESTED FROM: ::1 CALL: channel.software.clone(myuser, rhn-tools-rhel-x86_64-server-6, {label=rhn-tools-rhel-x86_64-server-6-satprep-2015-07-10, summary=Software channel cloned by Satprep, name=rhn-tools-rhel-x86_64-server-6 clone from 2015-07-10, parent_label=rhel-x86_64-server-6-satprep-2015-07-10}, false) CALLER: (myuser) TIME: 0.02 seconds
redstone.xmlrpc.XmlRpcFault: Channel name must be at least 6 characters long, begin with a letter and may contain only lowercase letters, hyphens ('-'), periods ('.'), underscores ('_'), numerals, spaces, parentheses and forward slashes ('/').
        at com.redhat.rhn.frontend.xmlrpc.BaseHandler.invoke(BaseHandler.java:186)
...

Improve verification log behavior

Currently satprep_prepare_maintenance.py completely rewrites the verification log when executed with -k (skip monitoring) or -K (skip snapshots). When preparing maintenance in batches (e.g. productive systems after development systems) the verification log is incomplete and/or contains invalid information.

The behavior needs to be changed like this:

If vlog exists: append information
If vlog exists and contains incorrect information (monitoring/snapshots): correct them
Append/correct information only within selected mode (monitoring only with -K or any, snapshots only with -k or any)

Implement better auth mechanism for satprep_schedule_downtime.py

Currenty the logon credentials for Nagios/Icinga/Thruk/Shinken are provided using -u / --username and -p / --password. This means that the credentials can be seen in the shell history. It would be better to implement the auth file mechanism that is also used for the Satellite authentification information

Adding channel label to name during patch freezing

When freezing multiple channels with satprep_patch_freeze.py it might be helpful to have a custom label also included in the Channel name - e.g. "rhel-x86_64-server-6 clone from 2015-11-16 (label)" instead of "rhel-x86_64-server-6 clone from 2015-11-16"