How to monitor custom metrics okd 3.11 / Problem to add another target about cluster-monitoring-operator HOT 16 CLOSED

@danilorsilva the prometheus serviceaccount cant access the namespace. The account should be system:serviceaccount:openshift-monitoring:prometheus-k8s

from cluster-monitoring-operator.

Can you check if the label selectors of your ServieMonitor are matching the labels of your Service?

from cluster-monitoring-operator.

Yes, the labels are okay,

from cluster-monitoring-operator.

Can you share the ServiceMonitor object? Did you also configure the endpoints path to scrape /prometheus? If one of these things in the ServiceMonitor do not match, it the target will be dropped during discovery.

from cluster-monitoring-operator.

@danilorsilva i have run in too the same problems while configuring heketi-metrics in the prometheus instance. Some tips to cross off your list if you are using the default Prometheus instance:

1. is the servicemonitoring in the correct space?
2. is the namespace/project correctly labeled to allow scraping
3. did you grant the prometheus service account the rights to use the selector in the servicemonitor?
4. What do the logs say from the prometheus pods and config-reloader?

from cluster-monitoring-operator.

@Rikbruggink

1 - The ServiceMonitor must be on the same namespace/project that the service that i want to scrape? Or the ServiceMonitor must be on openshit-monitoring namespace and only set the namespace target inside the servicemonitor file?
2 - I put openshift.io/cluster-monitoring=true label on the namespace.

from cluster-monitoring-operator.

The ServiceMonitor must be on openshit-monitoring namespace and only set the namespace target inside the servicemonitor file?

the servicemonitor must be in the openshift-monitoring namespace with a correct selector

from cluster-monitoring-operator.

What i did @Rikbruggink :

I have a namespace named test and i set the label openshift.io/cluster-monitoring=true to be monitored by the CMO.

Inside this namespace i have a pod and a service. This service has the label app: api-mobile and handle the requests to this pod.

I have created this servicemonitor:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    k8s-app: api-mobile-servicemonitor
  name: api-mobile-servicemonitor
  namespace: openshift-monitoring
spec:
  endpoints:
  - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
    interval: 30s
    port: 8080-tcp
    path: /prometheus
  namespaceSelector:
    matchNames:
    - test
  selector:
    matchLabels:
      app: api-mobile

The k8s-app is the label that i need to set on ServiceMonitor to be scraped bu Prometheus and load the configuration.

from cluster-monitoring-operator.

Did you grant the prometheus user rights to view the namespace?

from cluster-monitoring-operator.

No @Rikbruggink , how can i set this permission? Could you help me?

from cluster-monitoring-operator.

I can see my configuration on the "Configuration" option on Prometheus Web Interface, but i cannot see on "Targets" or "Service Discovery".

from cluster-monitoring-operator.

the command for that would be oc adm policy add-role-to-user view/edit/admin prometheus-serviceaccount. choose the role that you think is needed and use the correct serviceaccount

from cluster-monitoring-operator.

Okay, i will test and let you know. Thanks!

from cluster-monitoring-operator.

oc adm policy add-cluster-role-to-user cluster-admin system:serviceaccounts:prometheus-k8s

Without sucess.

from cluster-monitoring-operator.

I did a test with a Nginx + Service + ServiceMonitor on openshift-monitoring namespace, and it wotks. The problem is when i use another namespace.

from cluster-monitoring-operator.

@Rikbruggink my hero! It works!! Thank so much!

from cluster-monitoring-operator.