Connected to cluster, but do not get CPU, Pod or Memory Usage about kubegraf HOT 13 CLOSED

Hi Sergei, Thank you for the response. I can get to https://cluster_ip:10250/metrics and /metrics/cadvisor, however, I'm assuming I'm getting an "Unauthorized" at the kubegraf level. Right now, what I'm doing is using the token to communicate with the cluster. Should I generate the SSL credentials instead? Would that fix anything?

Hi Sergei, Thank you again for the response. I added the relabel configs and restarted my Prometheus server, however, the chart still is not picking up node (instance works in its place). Is there anything else I should do?

Hi Sergei, I did install Prometheus via helm, and when I went to install the operator component, I was told that it was already installed. I did add the relabel configs as per the previous issue and here's what my config looks like--sorry for all the Q's--but do you see anything here? There's a lot to look at, but it seems like what you pasted I have none of in my config file: global: scrape_interval: 1m scrape_timeout: 10s evaluation_interval: 1m alerting: alertmanagers: - kubernetes_sd_configs: - role: pod bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt insecure_skip_verify: false scheme: http timeout: 10s api_version: v1 relabel_configs: - source_labels: [__meta_kubernetes_namespace] separator: ; regex: prometheus replacement: $1 action: keep - source_labels: [__meta_kubernetes_pod_label_app] separator: ; regex: prometheus replacement: $1 action: keep - source_labels: [__meta_kubernetes_pod_label_component] separator: ; regex: alertmanager replacement: $1 action: keep - source_labels: [__meta_kubernetes_pod_container_port_number] separator: ; regex: null replacement: $1 action: drop rule_files: - /etc/config/rules - /etc/config/alerts scrape_configs: - job_name: prometheus honor_timestamps: true scrape_interval: 1m scrape_timeout: 10s metrics_path: /metrics scheme: http static_configs: - targets: - localhost:9090 - job_name: kubernetes-apiservers honor_timestamps: true scrape_interval: 1m scrape_timeout: 10s metrics_path: /metrics scheme: https kubernetes_sd_configs: - role: endpoints bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt insecure_skip_verify: true relabel_configs: - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name] separator: ; regex: default;kubernetes;https replacement: $1 action: keep - job_name: kubernetes-nodes honor_timestamps: true scrape_interval: 1m scrape_timeout: 10s metrics_path: /metrics scheme: https kubernetes_sd_configs: - role: node bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt insecure_skip_verify: true relabel_configs: - separator: ; regex: __meta_kubernetes_node_label_(.+) replacement: $1 action: labelmap - separator: ; regex: (.*) target_label: __address__ replacement: kubernetes.default.svc:443 action: replace - source_labels: [__meta_kubernetes_node_name] separator: ; regex: (.+) target_label: __metrics_path__ replacement: /api/v1/nodes/$1/proxy/metrics action: replace - separator: ; regex: __meta_kubernetes_node_label_kubernetes_io_hostname replacement: node action: labelmap - job_name: kubernetes-nodes-cadvisor honor_timestamps: true scrape_interval: 1m scrape_timeout: 10s metrics_path: /metrics scheme: https kubernetes_sd_configs: - role: node bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt insecure_skip_verify: true relabel_configs: - separator: ; regex: __meta_kubernetes_node_label_(.+) replacement: $1 action: labelmap - separator: ; regex: (.*) target_label: __address__ replacement: kubernetes.default.svc:443 action: replace - source_labels: [__meta_kubernetes_node_name] separator: ; regex: (.+) target_label: __metrics_path__ replacement: /api/v1/nodes/$1/proxy/metrics/cadvisor action: replace - separator: ; regex: __meta_kubernetes_node_label_kubernetes_io_hostname replacement: node action: labelmap - job_name: kubernetes-service-endpoints honor_timestamps: true scrape_interval: 1m scrape_timeout: 10s metrics_path: /metrics scheme: http kubernetes_sd_configs: - role: endpoints relabel_configs: - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape] separator: ; regex: "true" replacement: $1 action: keep - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme] separator: ; regex: (https?) target_label: __scheme__ replacement: $1 action: replace - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path] separator: ; regex: (.+) target_label: __metrics_path__ replacement: $1 action: replace - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port] separator: ; regex: ([^:]+)(?::\d+)?;(\d+) target_label: __address__ replacement: $1:$2 action: replace - separator: ; regex: __meta_kubernetes_service_label_(.+) replacement: $1 action: labelmap - source_labels: [__meta_kubernetes_namespace] separator: ; regex: (.*) target_label: kubernetes_namespace replacement: $1 action: replace - source_labels: [__meta_kubernetes_service_name] separator: ; regex: (.*) target_label: kubernetes_name replacement: $1 action: replace - source_labels: [__meta_kubernetes_pod_node_name] separator: ; regex: (.*) target_label: kubernetes_node replacement: $1 action: replace - job_name: prometheus-pushgateway honor_labels: true honor_timestamps: true scrape_interval: 1m scrape_timeout: 10s metrics_path: /metrics scheme: http kubernetes_sd_configs: - role: service relabel_configs: - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_probe] separator: ; regex: pushgateway replacement: $1 action: keep - job_name: kubernetes-services honor_timestamps: true params: module: - http_2xx scrape_interval: 1m scrape_timeout: 10s metrics_path: /probe scheme: http kubernetes_sd_configs: - role: service relabel_configs: - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_probe] separator: ; regex: "true" replacement: $1 action: keep - source_labels: [__address__] separator: ; regex: (.*) target_label: __param_target replacement: $1 action: replace - separator: ; regex: (.*) target_label: __address__ replacement: blackbox action: replace - source_labels: [__param_target] separator: ; regex: (.*) target_label: instance replacement: $1 action: replace - separator: ; regex: __meta_kubernetes_service_label_(.+) replacement: $1 action: labelmap - source_labels: [__meta_kubernetes_namespace] separator: ; regex: (.*) target_label: kubernetes_namespace replacement: $1 action: replace - source_labels: [__meta_kubernetes_service_name] separator: ; regex: (.*) target_label: kubernetes_name replacement: $1 action: replace - job_name: kubernetes-pods honor_timestamps: true scrape_interval: 1m scrape_timeout: 10s metrics_path: /metrics scheme: http kubernetes_sd_configs: - role: pod relabel_configs: - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape] separator: ; regex: "true" replacement: $1 action: keep - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path] separator: ; regex: (.+) target_label: __metrics_path__ replacement: $1 action: replace - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port] separator: ; regex: ([^:]+)(?::\d+)?;(\d+) target_label: __address__ replacement: $1:$2 action: replace - separator: ; regex: __meta_kubernetes_pod_label_(.+) replacement: $1 action: labelmap - source_labels: [__meta_kubernetes_namespace] separator: ; regex: (.*) target_label: kubernetes_namespace replacement: $1 action: replace - source_labels: [__meta_kubernetes_pod_name] separator: ; regex: (.*) target_label: kubernetes_pod_name replacement: $1 action: replace

Hi Sergei, I am running 0.18.0 node-exporter and 1.6.0 of prometheus-kube-state-metrics. I can make the the CPU gauge populate by changing the query to look like this: (sum(rate(container_cpu_usage_seconds_total{instance="$node"}[15m])) / sum(kube_node_status_allocatable_cpu_cores{node="$node"})) * 100 The material change is instance="$node"; just an FYI.

Hi Sergei, Yes, this fixed it (v1.2.0.5)! Thank you.