sum(container_memory_usage_bytes{...}) rule doubles values about kubernetes-mixin HOT 7 CLOSED

from kubernetes-mixin.

Hi @rrichardson; thanks for the issue! I'd be surprised if node_exporter is exporting container_* metrics, but cadvisor (embedded in the kubelet) exports metrics in a hierarchical fashion - and hence if we aggregate lower levels of the hierarchy with upper levels, we can get doubling...

We solve this by dropping lower levels at scrape time, see https://github.com/grafana/jsonnet-libs/blob/master/prometheus-ksonnet/lib/prometheus-config.libsonnet#L287

Could you confirm that image!="" to the rules fixes this for you? We should add that here, and see if we can have prometheus operator also drop those rules.

from kubernetes-mixin.

The Prometheus-Operator supports relabeling on the Endpoints of a ServiceMonitor:
https://github.com/coreos/prometheus-operator/blob/master/pkg/apis/monitoring/v1/types.go#L558

What we would need to do is to update this endpoint in the kube-prometheus stack:
https://github.com/coreos/prometheus-operator/blob/master/contrib/kube-prometheus/jsonnet/kube-prometheus/prometheus/prometheus.libsonnet#L280

Would you like to do a PR @rrichardson? Let me know, otherwise I can do the change too, but it's a nice little contribution if you want to.

from kubernetes-mixin.

@metalmatze Sure. I'll take a whack at it.

from kubernetes-mixin.

I still don't fully understand prometheus or the scope of cAdvisor and its hierarchy of metrics.

But shouldn't https://github.com/grafana/jsonnet-libs/blob/master/prometheus-ksonnet/lib/prometheus-config.libsonnet#L286-L292 already solve my problem?

It should be dropping container_.* where image =="" . when I filter out image = "" then my result is correct. But shouldn't those records not exist?

from kubernetes-mixin.

Hey @rrichardson ,

Thanks for documing your findings. I'm currently having the exact same problem with the metrics from cAdvisor not playing well with sum (for the exact same reason you opened this issue: the "total" (image="") is not filtered out as the relabeling rule does not exist in kube-prometheus).

I still don't fully understand prometheus or the scope of cAdvisor and its hierarchy of metrics.

But shouldn't https://github.com/grafana/jsonnet-libs/blob/master/prometheus-ksonnet/lib/prometheus-config.libsonnet#L286-L292 already solve my problem?

It should be dropping container_.* where image =="" . when I filter out image = "" then my result is correct. But shouldn't those records not exist?

I see that the PR on kube-prometheus was closed as this metric is not redundant and only Grafana dashboards should be fixed to filter it out.
However, I don't seem do find any patch on the dashboard side.

How did you ended up fixing the issue? Were the changes up-streamed and I'm missing something?

Edit:

To add a bit more context, here is an example of value I get in double:
https://github.com/kubernetes-monitoring/kubernetes-mixin/blob/master/dashboards/resources/pod.libsonnet#L106

Just like your issue, one is the Pod Total (container=""), the other is the container itself (container="manager"), container="POD" is already filtered out by the query.

from kubernetes-mixin.

I think this can be closed after #512 was merged.

from kubernetes-mixin.