Setting an ssh acknowledgement in Nagios for a switch that's down should cause that switch's servers to not be checked, but it looks like they get checked anyway. It creates noise in the alerts.

Currently, Rebot runs as salarcon on eb.measurementlab.net. It does so because it uses salarcon's .netrc file to authenticate to Nagios to get status from baselist, and salarcon's PLC account to run the drac.py command from the operator repo. Finally, because the script runs as salarcon, it uses [email protected] to send status emails.

Three changes should happen:

1. Make a system account to access Nagios, change the script to use the system account, OR cause the script to run baselist locally on eb rather than making a request to http://nagios.measurementlab.net/baseList?show_state=1&service_name=ssh&plugin_output=0&show_problem_acknowledged=1, etc. The decision was made to call a URL to keep the script portable, but maybe that trade-off doesn't pay off.

2. Use a different PLC account to run drac.py OR somehow make a local copy of DRAC passwords from the PLC database (or otherwise break the dependency on PLC for DRAC passwords).

3. Cause the script to run as a system user on eb, and make sure that user is allowed to post to the [email protected] Google Group. Current admins for the group are critzo, soltesz, salarcon, and kinkade.

The current alert email subject is the same every time, leaving all messages from Rebot to show up in a single thread in some mail clients.

Consider adding a unique value like the date/time (or whatever makes sense) to the subject line of Rebot alert emails.

Perhaps discuss in Ops standup what this format ought to look like and apply to all alert notifications from different systems?

This was observed on the week of Jun 1 on the SRE overview dashboard, when Rebot tried to reboot nodes at GRU01, ORD05 and IAD03 while these sites were offline.

I suspect this is a timing issue happening when Rebot runs its SwitchQuery when the switch hasn't been offline continuously for 15 minutes yet, but this needs to be investigated further.

The addition of the site label is currently being reviewed: m-lab/prometheus-support#466. After this is added to our configuration, we can remove the label_replace in the healthcheck query and use site directly.