KubePodNotReady is being fired for pods with restartPolicy=RestartNever about cluster-monitoring-operator HOT 8 CLOSED

Jobs failing doesn't mean the cluster is broken. There are lots of pods that can run to completion and exit with status code 0 as part of normal operation.

In this case, this is ci jobs running on the cluster that use restart=Never pods. We'll always have some level of failing builds on a cluster as well (openshift builds run as pods).

from cluster-monitoring-operator.

Can confirm - seeing this on some pods on free-int as well

from cluster-monitoring-operator.

I believe this would be tricky today as restartPolicy is not passed through via the kube-state-metrics I believe (/cc @brancz for verification).

What status are those job pods exactly in and what failure/exit mode is provoking those pods to be alerted? Looking at the alerting expression all job pods which exit with Succeeded should not fire any alerts:

sum by(namespace, pod) (kube_pod_status_phase{job="kube-state metrics",phase!~"Running|Succeeded"}) > 0

Wouldn't it make more sense to modify those job pods to exit with a Succeeded status?

from cluster-monitoring-operator.

It's likely that we could constrain this to only system namespaces kube-* / openshift-*, in which case the check is probably fine for now.

from cluster-monitoring-operator.

Yes we have limiting things to a list of namespaces already on our radar: kubernetes-monitoring/kubernetes-mixin#56

I think we're going to have to do this before code freeze, as it produces too much noise.

from cluster-monitoring-operator.

For the time being this was fixed by kubernetes-monitoring/kubernetes-mixin#70, we may be able to improve this a bit more in the future but for now it's not producing false positives.

We'll need to update the jsonnet dependency to get this.

from cluster-monitoring-operator.

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

from cluster-monitoring-operator.

The referenced mixins have been long merged, hence closing out.

from cluster-monitoring-operator.