weaveworks / alertdiff Goto Github PK

So we don't have to wait for the five minute Prometheus stale timeout.

We're healthy if:

• PushSources are being kept up to date
• PullSources aren't erroring too much
• We have successfully returned diff metrics recently

Our Cortex alerts have an additional monitor label which is causing alertdiff to report spurious differences; add an option to exclude specified labels from the comparison.

e.g. push/pull success/failures, latencies, GHC runtime stuffs

When alertdiff restarts, the timeseries it generates can get different pod name/instance/node labels resulting in duplicated details in the alerts. We should exclude these labels from the alert definitions:

IF max(alertdiff_spurious_alert) without (kubernetes_pod_name,instance,node) > 0

So that we can see what it's doing!

Because our reference Prometheus instance is not backed by a persistent store, it forgets firing alerts on restart which will result in spurious AlertDiffSpuriousAlert alerts. This isn't really a bug in alertdiff per se, but it is a problem which will reduce confidence in the alerts it raises.

See weaveworks/service-conf#2298.