techiescamp / kubernetes-prometheus Goto Github PK

View Code? Open in Web Editor NEW

370.0 9.0 508.0 62 KB

Kubernetes Manifest files for setting up Prometheus monitoring on the Kubernetes cluster.

Home Page: https://devopscube.com/setup-prometheus-monitoring-on-kubernetes/

prometheus kubernetes grafana monitoring alertmanager

kubernetes-prometheus's Introduction

👋 Hi, I’m @techiescamp

kubernetes-prometheus's People

Stargazers

Watchers

Forkers

devopsevd richmondwang eggsy84 jimmy-chuang hyperpilotio cwolfinger datle79 skashan-ali vkuznet karthiknew balugeorge agungxtwo russiancook jacky1104 xuefli zhoshen rhyspowell gabrielsagnard wudongyin juniorjbn shashi198 bashb1n chriswiggins ziyanakthar hardeep18 avaussant librannk renatofenrir remche muddanas camanducci chbharathkumar 4ndr4s ifmorrison nagarameshreddy tmack91 raavula dhavlev kamaleswarsreedharan 1235gopalt benapetr gitsoso akoranne ramimmadi sriharichava michaelsync lvyingde51 anilkumarkoduri shashank-org ibronit meashishk kapilgarg83 g0194776 coderwangke flynn-z ssongjunxian ganasubrgit parvathanenisrinu angryjohnnie cnivasvemula boriphuth keffadi sensedata1 gloriapg qh-hale kigromov cableman aalishe lerndevops sameeraksc ravikiran529 dashanhust wackxu sunlove123 cesigit modi1492 05kritika jawad8106 danitfk binnyrs gurram99143 5g-media nickopolar gokcerbelgusen seeker815 laidig karthikvee ramesh2631 tymoteuszgach y0zg ankitapradhan7996 raja1090 putugandi maxmiles39 biju-vinoth narendra311777 ankitsoni570 subject33 romemoore munnysri

kubernetes-prometheus's Issues

kubernetes-service-endpoints is down

kubernetes-service-endpoints is down. Connection refused. Using Exposing Prometheus As A Service, e.g. NodePort with the following annotations:
prometheus.io/scrape: 'true'
prometheus.io/path: /
prometheus.io/port: '8080'

Any hint of what might be wrong ? Also pods are not showing in Prometheus. Using 2.2.1 and 2.3.2 version of the Prometheus image shows same results.

Thanks for repo.

testing yaml

...

....

how can we setup normal role and normal role binding for prometheus setup

Hi can we use normal role for prometheus setup if we use how to use non-resource url [/metrics] under namespace scoped role and wn i setup using normal role and normal role binding i dont see any metrics except kube-state?

Upgrade Prometheus Image

configmap incorrect format

when apply to k8s,it incorrect format like this：

root@demo ~]# kubectl -n monitor get cm prometheus-server-conf -oyaml
apiVersion: v1
data:
  prometheus.rules: |-
    groups:
    - name: devopscube demo alert
      rules:
      - alert: High Pod Memory
        expr: sum(container_memory_usage_bytes) > 1
        for: 1m
        labels:
          severity: slack
        annotations:
          summary: High Memory Usage
  prometheus.yml: "global:\n  scrape_interval: 5s\n  evaluation_interval: 5s\nrule_files:\n
    \ - /etc/prometheus/prometheus.rules\nalerting:\n  alertmanagers:\n  - scheme:
    http\n    static_configs:\n    - targets:\n      - \"alertmanager.monitoring.svc:9093\"\n\nscrape_configs:\n
    \ - job_name: 'node-exporter'\n    kubernetes_sd_configs:\n      - role: endpoints\n
    \   relabel_configs:\n    - source_labels: [__meta_kubernetes_endpoints_name]\n
    \     regex: 'node-exporter'\n      action: keep\n  \n  - job_name: 'kubernetes-apiservers'\n\n
    \   kubernetes_sd_configs:\n    - role: endpoints\n    scheme: https\n\n    tls_config:\n
    \     ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt\n    bearer_token_file:
    /var/run/secrets/kubernetes.io/serviceaccount/token\n\n    relabel_configs:\n
    \   - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name,
    __meta_kubernetes_endpoint_port_name]\n      action: keep\n      regex: default;kubernetes;https\n\n
    \ - job_name: 'kubernetes-nodes'\n\n    scheme: https\n\n    tls_config:\n      ca_file:
    /var/run/secrets/kubernetes.io/serviceaccount/ca.crt\n    bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token\n\n
    \   kubernetes_sd_configs:\n    - role: node\n\n    relabel_configs:\n    - action:
    labelmap\n      regex: __meta_kubernetes_node_label_(.+)\n    - target_label:
    __address__\n      replacement: kubernetes.default.svc:443\n    - source_labels:
    [__meta_kubernetes_node_name]\n      regex: (.+)\n      target_label: __metrics_path__\n
    \     replacement: /api/v1/nodes/${1}/proxy/metrics\n  \n  - job_name: 'kubernetes-pods'\n\n
    \   kubernetes_sd_configs:\n    - role: pod\n\n    relabel_configs:\n    - source_labels:
    [__meta_kubernetes_pod_annotation_prometheus_io_scrape]\n      action: keep\n
    \     regex: true\n    - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]\n
    \     action: replace\n      target_label: __metrics_path__\n      regex: (.+)\n
    \   - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]\n
    \     action: replace\n      regex: ([^:]+)(?::\\d+)?;(\\d+)\n      replacement:
    $1:$2\n      target_label: __address__\n    - action: labelmap\n      regex: __meta_kubernetes_pod_label_(.+)\n
    \   - source_labels: [__meta_kubernetes_namespace]\n      action: replace\n      target_label:
    kubernetes_namespace\n    - source_labels: [__meta_kubernetes_pod_name]\n      action:
    replace\n      target_label: kubernetes_pod_name\n  \n  - job_name: 'kube-state-metrics'\n
    \   static_configs:\n      - targets: ['kube-state-metrics.kube-system.svc.cluster.local:8080']\n\n
    \ - job_name: 'kubernetes-cadvisor'\n\n    scheme: https\n\n    tls_config:\n
    \     ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt\n    bearer_token_file:
    /var/run/secrets/kubernetes.io/serviceaccount/token\n\n    kubernetes_sd_configs:\n
    \   - role: node\n\n    relabel_configs:\n    - action: labelmap\n      regex:
    __meta_kubernetes_node_label_(.+)\n    - target_label: __address__\n      replacement:
    kubernetes.default.svc:443\n    - source_labels: [__meta_kubernetes_node_name]\n
    \     regex: (.+)\n      target_label: __metrics_path__\n      replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor\n
    \ \n  - job_name: 'kubernetes-service-endpoints'\n\n    kubernetes_sd_configs:\n
    \   - role: endpoints\n\n    relabel_configs:\n    - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]\n
    \     action: keep\n      regex: true\n    - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]\n
    \     action: replace\n      target_label: __scheme__\n      regex: (https?)\n
    \   - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]\n
    \     action: replace\n      target_label: __metrics_path__\n      regex: (.+)\n
    \   - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]\n
    \     action: replace\n      target_label: __address__\n      regex: ([^:]+)(?::\\d+)?;(\\d+)\n
    \     replacement: $1:$2\n    - action: labelmap\n      regex: __meta_kubernetes_service_label_(.+)\n
    \   - source_labels: [__meta_kubernetes_namespace]\n      action: replace\n      target_label:
    kubernetes_namespace\n    - source_labels: [__meta_kubernetes_service_name]\n
    \     action: replace\n      target_label: kubernetes_name"
kind: ConfigMap

Setting up grafana on Kubernetes cluster

hey @bibinwilson @devopscube - have you documented any steps for setting up grafana on kubernetes cluster ?

kubernetes-service-endpoints - connection refused under targets

I have installed as per your files/instructions however one endpoint is showing as down. Please let me know if anything can be done to correct this.

Service endpoint are not getting discovered. Required a config change in target configuration

Prometheus Image update

Prometheus service endpoint name can't resolve by coredns service

Hi Bibinwilson,
I deploy kubernetes-prometheus repository on my local cluster then below error occurred in Prometheus-dashboard.

Prometheus UI error in targets:

Get "http://kube-state-metrics.kube-system.svc.cluster.local:8080/metrics": dial tcp: lookup kube-state-metrics.kube-system.svc.cluster.local on 10.96.0.10:53: no such host

Expose `kubernetes_pod_name` label to `kubernetes-service-endpoints` job.

I believe, lot of applications might wanna have the kuberenetes_pod_name label exposed. Since for each kuberenetes_name there could be multiple replica sets. Having this by default should be appropriate in my opinion.

Add kube state metrics endpoint to prometheus config

No graphs

Hi,
I read your blog post and followed all instructions. The prometheus service is up and running but I don't see any graphs. The only thing under Targets is a checkmark "Only unhealthy jobs". In a graph page I don't see any metrics either, e.g. if I enter kubernetes-nodes, I see "Load time" but nothing more.

My first question is:

does it monitor apps/services in monitoring namespace or in all namespaces?
I have few apps/pods/services running default namespace and I wonder may be I need to deploy prometheus in this specific namespace?
the configuration file contains config for several endpoints and I wonder if something needs to be adjusted for specific cluster setup.

Thanks,
Valentin.

Can I have a look at your prometheus.yml file?

Adding Target to Prometheus using ConfigMap

Hi everyone,
i trying to add some targets to Prometheus instance on local k8s cluster. First i installed prometheus-operator using helm chart. After that, i created a ServiceMonitor:
prometheusservicemonitor.txt

And to monitor my netcore application, which is running also as a service of type ClusterIP and Deployment,
netcoreapp-service.txt

netcoreapp-deployment.txt

As next, i created a Prometheus instance in this way:
prometheusinstance.txt

I applied both in kubectl and set my namespace. I restarted Prometheus instances using:
kubectl scale --replicas=0 deployment --all and kubectl scale --replicas=1 deployment --all

Even my service has the annotations prometheus: scrape, path and port, i can't see my app in the targets of prometheus.
I tried using this ConfigMap:
prometheus-configmap.txt

But in all cases i can't see my app as a target or in the configuration as job_name in Prometheus.
I'm new in Kubernetes/Prometheus World, it would be great if some can tell me what i did wrong or to provide me a concret example with adding target or overwriting Prometheus configs installed with helm

prometheus-service is down

Hi,

First of all, thanks for this repository, it is very easy to understand and very practical. Secondly, when I deploy Prometheus in my Kubernetes environment (kubeadm), I have an error with prometheus-service. Here you are an screenshot:

Thanks in advance,

Regards

update apiVersion for prometheus-deployment.yaml

several APIs were deprecated in k8s v1.16, including the one in the deployment.

Just need to update the ap1Version to "apps/v1" and add a selector in the spec

https://kubernetes.io/blog/2019/07/18/api-deprecations-in-1-16/