kubernetes service discovery about prometheus-pve-exporter HOT 9 CLOSED

Any idea how this could work?

Regrettably I cannot help with that directly. I recall that snmp_exporter is using the same pattern (i.e., a target url parameter). So you might get pointers on how to configure service discovery by looking for examples of snmp_exporter deployments using kubernetes.

from prometheus-pve-exporter.

Just guessing wildly, do you maybe set metrics_path: /pve?target=x.x.x.x in prometheus.yml? That would at least explain the percent encoding.

from prometheus-pve-exporter.

No, I don't set anything in prometheus.yml. This is discovered through yaml file of your container, which prometheus on k8s auto-detects out of the box using annotations on the pod template. The below config for your container ended up being properly auto-detected by prometheus, except for the ? being escaped.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: proxmox-exporter
  namespace: prometheus
spec:
  replicas: 1
  selector:
    matchLabels:
      app: proxmox-exporter
  template:
    metadata:
      labels:
        app: proxmox-exporter
      annotations:
        prometheus.io/scrape: 'true'
        prometheus.io/path: "/pve?target=1.2.3.4"
        prometheus.io/port: '9221'
        prometheus.io/scheme: 'http'
    spec:
      containers:
        - name: proxmox-exporter
          image: prompve/prometheus-pve-exporter
          volumeMounts:
            - name: proxmox-exporter-config
              mountPath: /etc/pve.yml
              subPath: pve.yml
      volumes:
        - name: proxmox-exporter-config
          configMap:
            name: proxmox-exporter-config

When I used the prometheus-operator helm chart, it supported a serviceMonitor yaml file for discovery, where you could pass this data slightly differently, but that doesnt seem to work with annotations.

  - port: proxmox-exporter
    path: /pve
    params:
      target:
        - 192.168.1.12
    interval: 15s

Would it be possible to maybe allow for a second routing options like http://foo.bar/pve/1.2.3.4 , that might solve the issue. But I totally understand if thats too much hassle, and like I said, I just ended up using a scraper config so there is not an issue as much.

from prometheus-pve-exporter.

Right. I guess that prometheus.io/path: "/pve?target=1.2.3.4" annotation ended up in metrics_path. Could you try and set the annotation to prometheus.io/path: "/pve" and instead inject the param via job configuration in prometheus.yml. I.e.:

  - job_name: "my-prometheus-pve-job"
    kubernetes_sd_configs:
      - role: endpoints
    params:
      target:
        - 192.168.1.12

from prometheus-pve-exporter.

I might be missing the whole picture regarding the kube setup, but I bet there is a way to supply url parameters either via annotations or job configuration to the exporter. There is also blackbox_exporter which has the same requirement and I bet that people are running that in kubernetes as well.

from prometheus-pve-exporter.

I looked at the blackbox exporter and in the prometheus scraper config it specifically allows for a target to be set through a k8s label:

  - source_labels: [__address__]
    target_label: __param_target

The default service/pod scrapers don't, because I guess most metrics targets don't need params. So I'd have to either make my own kubernetes_sd_config specifically for pve_exporter (then I might as well keep my current pve scraper), or see if I can modify the existing sd scraper to allow for (arbitrary) params.

It's not really a big deal, it just seems that out of the box this specific container can't be service-discovered as-is by a default prometheus k8s install.

from prometheus-pve-exporter.

Ok, I managed to fix this in the existing kubernetes_sd scraper by adding the following:

          - action: labelmap
            regex: __meta_kubernetes_pod_annotation_prometheus_io_param_(.+)
            replacement: __param_$1

I'll see if the helm chart for prometheus is interested in a PR.

from prometheus-pve-exporter.

Just FYI, I created a PR for prometheus helm chart to address this issue. We'll see if they merge it, at least it was approved.

prometheus-community/helm-charts#1298

from prometheus-pve-exporter.

Oh cool, thanks!

from prometheus-pve-exporter.