Adding slack notifications about chaoskube HOT 6 OPEN

Could we add a prometheus metric for that?

I often saw some "binary" metrics that you are supposed to "query" via labels and then either return 0 or 1, e.g.

up{application="dnsmasq-node", ...} 1
up{instance="ip-172-31-8-124.eu-central-1.compute.internal", ...} 1
...

We could expose something similar for chaoskube, e.g.

chaoskube_active 1 # or 0

It's not comparable to a notification but at least you would have a structured way to find out whether chaoskube is in an active period right now.

from chaoskube.

Not at all. I just wanted to provide an inferior alternative that might save us some effort.

I'm fine with adding slack support. How about putting it in a separate package and hiding it behind a nice Notifer interface?

Let me know how I can help. I'm looking forward to seeing what you come up with.

from chaoskube.

I'll add that with the current codebase there is no way to know that we've "entered" a chaos period (a period with no exclusions) or "exit" (a period that excluded).
The only way to understand that as I see it, is to check if the previous interval was in exclusion time.

from chaoskube.

We can definitely add a prometheus metric for that, but that would serve us for alerts and monitoring.

I would still like to make a slack notification for that, so the team will be actively aware when a chaos is running, and be prepared if action is required.

Do you prefer to avoid adding slack capabilities to this tool?

from chaoskube.

Great, I'll try to work on that soon and I hope to return with a PR 👍

from chaoskube.

I often use alertmanager to send notifications to different channels, including Slack. With this approach, a Prometheus metric is enough and there is a clear separation of responsibilities.

from chaoskube.