run Prometheus no node about kubernetes-app HOT 5 CLOSED

You can run your Prometheus external to the the cluster. You can find more information on how to do that here

You will have to update the kubernetes sd config for prometheus to access your cluster.

More information on that can be found here: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#kubernetes_sd_config

from kubernetes-app.

msg="Cannot create Kubernetes discovery" err="unable to load in-cluster configuration, KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT must be defined"

from kubernetes-app.

scrape_configs:

• job_name: 'kubernetes-kubelet'
  scheme: https
  tls_config:
  ca_file: /opt/prometheus/k8s_cert_pre/ca.crt
  insecure_skip_verify: true
  bearer_token_file: /opt/prometheus/k8s_cert_pre/token
  kubernetes_sd_configs:
  • role: node
    relabel_configs:
  • action: labelmap
    regex: _meta_kubernetes_node_label(.+)
  • target_label: address
    replacement: 126.201.27.100:443 (hapster IP)
  • source_labels: [__meta_kubernetes_node_name]
    regex: (.+)
    target_label: metrics_path
    replacement: /api/v1/nodes/${1}/proxy/metrics
• job_name: 'kubernetes-cadvisor'
  scheme: https
  tls_config:
  ca_file: /opt/prometheus/k8s_cert_pre/ca.crt
  insecure_skip_verify: true
  bearer_token_file: /opt/prometheus/k8s_cert_pre/token
  kubernetes_sd_configs:
  • role: node
    relabel_configs:
  • action: labelmap
    regex: _meta_kubernetes_node_label(.+)
  • target_label: address
    replacement: 126.201.27.100:443 (hapster IP)
  • source_labels: [__meta_kubernetes_node_name]
    regex: (.+)
    target_label: metrics_path
    replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor
• job_name: 'kubernetes-kube-state'
  kubernetes_sd_configs:
  • role: pod
    relabel_configs:
  • action: labelmap
    regex: _meta_kubernetes_pod_label(.+)
  • source_labels: [__meta_kubernetes_namespace]
    action: replace
    target_label: kubernetes_namespace
  • source_labels: [__meta_kubernetes_pod_name]
    action: replace
    target_label: kubernetes_pod_name
  • source_labels: [__meta_kubernetes_pod_label_grafanak8sapp]
    regex: .true.
    action: keep
  • source_labels: ['__meta_kubernetes_pod_label_daemon', '__meta_kubernetes_pod_node_name']
    regex: 'node-exporter;(.*)'
    action: replace
    target_label: nodename

from kubernetes-app.

@bernapa0 You are missing the api_server config.

Excerpt from the documentation linked in my first comment:

# The API server addresses. If left empty, Prometheus is assumed to run inside
# of the cluster and will discover API servers automatically and use the pod's
# CA certificate and bearer token file at /var/run/secrets/kubernetes.io/serviceaccount/.
[ api_server: <host> ]

Because you are not setting the api_server config, the Prometheus is running as if it is in the cluster, which it is not.

from kubernetes-app.

I am trying to replicate the setup. I am adding below config to my external prometheus server.

• job_name: 'kubernetes-pods'
  scheme: https
  kubernetes_sd_configs:

  • role: endpoints
    api_server: 'url:443'

    bearer_token:

    #tls_config:
    #ca_file: /etc/prometheus/ca.cert
    #insecure_skip_verify: true
    #bearer_token_file: /etc/prometheus/bearer-token

  relabel_configs:

  • action: labelmap
    regex: _meta_kubernetes_pod_label(.+)
  • source_labels: [__meta_kubernetes_namespace]
    action: replace
    target_label: kubernetes_namespace
  • source_labels: [__meta_kubernetes_pod_name]
    action: replace
    target_label: kubernetes_pod_name

What iis the correct way of adding tls config or the bearer token.

If i use bearer token should i still give cert ?
ALso could this be due to the default user being used ?

I get below error message.

level=error ts=2020-11-20T19:54:42.790Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg="/app/discovery/kubernetes/kubernetes.go:263: Failed to list *v1.Pod: the server rejected our request for an unknown reason (get pods)"

from kubernetes-app.