Comments (5)
You can run your Prometheus external to the the cluster. You can find more information on how to do that here
You will have to update the kubernetes sd config for prometheus to access your cluster.
More information on that can be found here: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#kubernetes_sd_config
from kubernetes-app.
msg="Cannot create Kubernetes discovery" err="unable to load in-cluster configuration, KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT must be defined"
from kubernetes-app.
scrape_configs:
- job_name: 'kubernetes-kubelet'
scheme: https
tls_config:
ca_file: /opt/prometheus/k8s_cert_pre/ca.crt
insecure_skip_verify: true
bearer_token_file: /opt/prometheus/k8s_cert_pre/token
kubernetes_sd_configs:- role: node
relabel_configs: - action: labelmap
regex: _meta_kubernetes_node_label(.+) - target_label: address
replacement: 126.201.27.100:443 (hapster IP) - source_labels: [__meta_kubernetes_node_name]
regex: (.+)
target_label: metrics_path
replacement: /api/v1/nodes/${1}/proxy/metrics
- role: node
- job_name: 'kubernetes-cadvisor'
scheme: https
tls_config:
ca_file: /opt/prometheus/k8s_cert_pre/ca.crt
insecure_skip_verify: true
bearer_token_file: /opt/prometheus/k8s_cert_pre/token
kubernetes_sd_configs:- role: node
relabel_configs: - action: labelmap
regex: _meta_kubernetes_node_label(.+) - target_label: address
replacement: 126.201.27.100:443 (hapster IP) - source_labels: [__meta_kubernetes_node_name]
regex: (.+)
target_label: metrics_path
replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor
- role: node
- job_name: 'kubernetes-kube-state'
kubernetes_sd_configs:- role: pod
relabel_configs: - action: labelmap
regex: _meta_kubernetes_pod_label(.+) - source_labels: [__meta_kubernetes_namespace]
action: replace
target_label: kubernetes_namespace - source_labels: [__meta_kubernetes_pod_name]
action: replace
target_label: kubernetes_pod_name - source_labels: [__meta_kubernetes_pod_label_grafanak8sapp]
regex: .true.
action: keep - source_labels: ['__meta_kubernetes_pod_label_daemon', '__meta_kubernetes_pod_node_name']
regex: 'node-exporter;(.*)'
action: replace
target_label: nodename
- role: pod
from kubernetes-app.
@bernapa0 You are missing the api_server
config.
Excerpt from the documentation linked in my first comment:
# The API server addresses. If left empty, Prometheus is assumed to run inside
# of the cluster and will discover API servers automatically and use the pod's
# CA certificate and bearer token file at /var/run/secrets/kubernetes.io/serviceaccount/.
[ api_server: <host> ]
Because you are not setting the api_server
config, the Prometheus is running as if it is in the cluster, which it is not.
from kubernetes-app.
I am trying to replicate the setup. I am adding below config to my external prometheus server.
-
job_name: 'kubernetes-pods'
scheme: https
kubernetes_sd_configs:-
role: endpoints
api_server: 'url:443'bearer_token:
#tls_config:
#ca_file: /etc/prometheus/ca.cert
#insecure_skip_verify: true
#bearer_token_file: /etc/prometheus/bearer-token
relabel_configs:
- action: labelmap
regex: _meta_kubernetes_pod_label(.+) - source_labels: [__meta_kubernetes_namespace]
action: replace
target_label: kubernetes_namespace - source_labels: [__meta_kubernetes_pod_name]
action: replace
target_label: kubernetes_pod_name
-
What iis the correct way of adding tls config or the bearer token.
If i use bearer token should i still give cert ?
ALso could this be due to the default user being used ?
I get below error message.
level=error ts=2020-11-20T19:54:42.790Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg="/app/discovery/kubernetes/kubernetes.go:263: Failed to list *v1.Pod: the server rejected our request for an unknown reason (get pods)"
from kubernetes-app.
Related Issues (20)
- InfluxDB/Telegraf Support
- Panel Plugin missing for grafana-kubernetes-app plugin HOT 2
- node_filesystem variable outdate HOT 1
- Adding datasource prometheus in grafana HTTP Error Not Found
- Error while initializing promtheus configmap, Node Exporter and Kube-State Metrics via Grafana
- New version of kube-state-metrics shows deprecations upcoming which are used in k8s-cluster dashboard
- Install command is wrong in https://grafana.com/grafana/plugins/grafana-kubernetes-app HOT 1
- public/plugins/grafana-kubernetes-datasource/img/logo.svg not found
- No Data Points Collected After Successful Deployment? HOT 3
- Memory Metrics are doubled Grafana HOT 1
- Is this app abandoned? HOT 5
- Why does this plugin ask me for user/passwd (basic auth)?
- Add a new cluster only gives Basic auth option HOT 1
- How to configure kubernetes cluster HOT 5
- Datasource is hardcoded for `v1beta1` api HOT 3
- invalid documentation
- Grafana not showing Pod/Container name in dashboard
- Connecting to K8s cluster HOT 1
- Pod filtering panel broken HOT 1
- Unsigned external plugin HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from kubernetes-app.