Cluster Disk Usage is broken since there is no nodename label about kubernetes-app HOT 10 OPEN

For Cluster Disk Usage gauge I changed the query to:

sum (container_fs_usage_bytes{device=~"^/dev/[nsx][v][bdm][a-z].+$",id="/",instance=~"^$node"}) / sum (container_fs_limit_bytes{device=~"^/dev/[nsx][v][bdm][a-z].+$",id="/",instance=~"^$node"})

and for me seems ok.

I changed also the queries for the Cluster Disk Capacity:
usage:

sum (container_fs_usage_bytes{device=~"^/dev/[xv][sv]d[a-z][1-9]$",id="/",instance=~"$node"})

limit:

sum (container_fs_limit_bytes{device=~"^/dev/[xv][sv]d[a-z][1-9]$",id="/",instance=~"$node"})

*I use kubernetes 1.9.6

from kubernetes-app.

I'm using this config for Prometheus, which contains nodename fix. The only difference from default config is that

          - source_labels: [__meta_kubernetes_pod_node_name]
            action: replace
            target_label: nodename

added to kubernetes-service-endpoints. I'm using Prometheus nodeExporter v0.15.2..

You can find more details here https://scalablesystem.design/categories/ds101/

from kubernetes-app.

Same issue

from kubernetes-app.

same issue

from kubernetes-app.

same issue

from kubernetes-app.

Same here.

Any suggested workarounds?

from kubernetes-app.

I tried a bunch of different queries and got results, but this seems more accurate. Thanks @vpistis . I will play around with the queries some more and let you know how I go. Cheers

from kubernetes-app.

awesome thanks @artyomboyko ,

Including this:

          - source_labels: [__meta_kubernetes_pod_node_name]
            action: replace
            target_label: nodename

To the prometheus kubernetes-service-endpoints config works like a charm as well. Thanks for this!

from kubernetes-app.

I had the same problem (no disk usage info in Grafana), but in my case it was version of NodeExporter.
I was using 0.17, but in 0.16 they've changed node_filesystem_size -> node_filesystem_size_bytes.
Rolled back to 0.15.2 and now it works.

from kubernetes-app.

If you're using a newer version of node-exporter (version 0.16 or 0.17), you can use job rules defines on upgrade guide: https://github.com/prometheus/node_exporter/blob/master/docs/V0_16_UPGRADE_GUIDE.md#use-recording-rules
This way older and newer metrics renamed both exists (but it take more space on prometheus).

from kubernetes-app.