node-exporter can't be configured to tolerate a taint about cluster-monitoring-operator HOT 9 CLOSED

CMO image on quay.io is updated after each merge to master. At the time of writing image with tag latest was updated 3 hours ago.

Source: https://quay.io/repository/openshift/origin-cluster-monitoring-operator?tab=tags

from cluster-monitoring-operator.

Yes, exactly. The operator will reconcile every few seconds and override those changes. What would you like to change for the taint to be recognized?

from cluster-monitoring-operator.

I was thinking you would just add it to the cluster-monitoring-config under the node-exporter section, similar to the node selector sections, except for tolerations.

for example:

prometheusOperator:
  baseImage: registry.redhat.io/openshift3/ose-prometheus-operator
  prometheusConfigReloaderBaseImage: registry.redhat.io/openshift3/ose-prometheus-config-reloader
  configReloaderBaseImage: registry.redhat.io/openshift3/ose-configmap-reloader
  nodeSelector:
    node-role.kubernetes.io/infra: "true"
prometheusK8s:
  baseImage: registry.redhat.io/openshift3/prometheus
  nodeSelector:
    node-role.kubernetes.io/infra: "true"
  externalLabels:
    cluster: console-a.openshift.example.com
alertmanagerMain:
  baseImage: registry.redhat.io/openshift3/prometheus-alertmanager
  nodeSelector:
    node-role.kubernetes.io/infra: "true"
nodeExporter:
  baseImage: registry.redhat.io/openshift3/prometheus-node-exporter
  tolerations:
    - key: "example-taint"
      operator: "Equal"
      value: "true"
      effect: "NoSchedule"
grafana:
  baseImage: registry.redhat.io/openshift3/grafana
  nodeSelector:
    node-role.kubernetes.io/infra: "true"
kubeStateMetrics:
  baseImage: registry.redhat.io/openshift3/ose-kube-state-metrics
  nodeSelector:
    node-role.kubernetes.io/infra: "true"
kubeRbacProxy:
  baseImage: registry.redhat.io/openshift3/ose-kube-rbac-proxy
auth:
  baseImage: registry.redhat.io/openshift3/oauth-proxy

or ignore taints altogether for the node-exporter if thats somehow easily achievable (and won't cause problems, this could be a bad idea though, not sure).

from cluster-monitoring-operator.

I believe node-exporter has to tolerate any taints:

tolerations:
  - operator: "Exists"

from cluster-monitoring-operator.

Just ran into this issue with tainted nodes as well: These will not have the node-exporter pod scheduled on them and thus no metric of them is available...

Is there any workaround that you know of?

from cluster-monitoring-operator.

Seems like this is fixed in master and release-3.11 branch roughly three weeks ago (file: daemonset.yaml ) : fe4ef56 and 3b4ebe6

However, the last release on quay.io seems to be 9 months old.
Is there a possibility of creating a new release of this and pushing it to Quay?

from cluster-monitoring-operator.

Closing as this is already fixed on master branch.

from cluster-monitoring-operator.

@paulfantom Is the 3.11 Image not being updated from the release-3.11 branch?
The labels for the quay.io Image for 3.11 points to the commit 10e4317 which is from February this year.

from cluster-monitoring-operator.

@padyx Yes, that's a bug and we are working on a solution.

from cluster-monitoring-operator.