Why do my changes to the prometheus rules only work in a short time? about cluster-monitoring-operator HOT 6 CLOSED

@squat You are right, this check is absolutely right. I will silent this check for resource="imagestreamimports" in alertmanager. Thanks for your fast response.

from cluster-monitoring-operator.

I'm also interested in overwrite default configuration, because of openshift/origin#21508

from cluster-monitoring-operator.

Hi @wenj1104, the resources created by the Cluster Monitoring Operator are entirely managed by the operator. That means that all changes made to them will always be overwritten within a few minutes once the controller re-syncs. This reconciliation is central to the operator pattern and ensures that the resources are always in a known good state and that we can automatically update them at a latter time without worrying about merge-conflicts.

If you want to change an existing rule, as it seems you do, then we'll need to make the change upstream and ship the change with the Cluster Monitoring Operator. Note that many of the rules shipped by this operator are developed outside of this repo and pulled in from different projects. In the case of this particular rule, I'd suggest filing an issue against the Kubernetes-Monitoring project [0], which is where the rule is defined. Chances are others may agree that the rule needs changing and that we'll all benefit from the patch :)

Alternatively, if you want to define an alert at a lower threshold, as you are doing here, then a workaround could be to add a new PrometheusRule objects that specifies this expression. In this case you'd receive an alert both at >90 pods and >100 pods. (please ignore this if that was just an example and not the change you actually needed)

[0] https://github.com/kubernetes-monitoring/kubernetes-mixin/blob/master/alerts/system_alerts.libsonnet#L65-L77

from cluster-monitoring-operator.

@Reamer the same advice applies regarding any changes to the default shipped alerting rules. In the case of your particular alert, the source is also defined in Kubernetes-Monitoring [0]. Please feel free to open a new issue there if you think the alerting rule should be changed for the community and in default OpenShift. From the information in openshift/origin#21508 it looks like the rule is alerting correctly and that the API server latency is indeed too high, so it may not be a desirable change. In any case, let's discuss more over at Kuberentes-Monitoring. If you simply want to ignore that alerting rule, another option would be to silence the alert in Alertmanager.

[0] https://github.com/kubernetes-monitoring/kubernetes-mixin/blob/master/alerts/system_alerts.libsonnet#L78-L90

from cluster-monitoring-operator.

@squat Okay,I get it.Your explanation is quite clear.I decided not to change the default rule.Thanks for your fast response.

from cluster-monitoring-operator.

@wenj1104 ok sounds good. I’ll close this issue for now but please file a new one if you have any more problems :)

from cluster-monitoring-operator.