Why not use pod annotations to scrape metrics? about clickhouse-operator HOT 15 CLOSED

@jordivilaseca thanks for clarification. You'd like to add custom labels/annotations to pods, managed by clickhouse-operator. Here is another issue #281 about custom labels. We'll consider the best way to add custom labels.
However, clickhouse-operator allows to specify custom labels/annotations for CHI object as ususal:

apiVersion: "clickhouse.altinity.com/v1"
kind: "ClickHouseInstallation"
metadata:
  name: "clickhouse-installation-max"
  labels:
YOUR LABELS GO HERE

due to CHI being specified as:

type ClickHouseInstallation struct {
	metav1.TypeMeta   `json:",inline"`
	metav1.ObjectMeta `json:"metadata,omitempty" yaml:"metadata"`
	Spec              ChiSpec   `json:"spec"     yaml:"spec"`
	Status            ChiStatus `json:"status"`
}

and clickhouse-operator applies custom labels from chi to all managed pods. This feature is not properly documented ATM, and it is like in development state, but you can test whether it works (probably as temp workaround) for you.

from clickhouse-operator.

Ok, that makes sense, we'll take a look into this. Thanks for suggestion!

from clickhouse-operator.

prometheus-operator is able to automatically discover pods to scrape metrics based on SD.
At least the one installed with helm chart
https://github.com/helm/charts/blob/master/stable/prometheus/values.yaml#L1478

Dear @Nklya you're confusing prometheus and prometheus-operator
according to https://github.com/coreos/prometheus-operator/tree/master/Documentation/user-guides, coreos prometheus-operator CRD can scrape metrics via ServiceMonitor or PodMonitor CRD which based only on LabelSelector mechanism and annotations can't be applied

But if you install prometheus via helm chart, you can use annotations for clickhouse-opearator deployment

I add pull request with this ability

from clickhouse-operator.

@vinayak-mehta you should use
.spec.templates.podTemplates.metadata.annotation instead of .metadata.annotation

in your case it will something like this

apiVersion: "clickhouse.altinity.com/v1"
kind: "ClickHouseInstallation"
metadata:
  name: "clickhouse"
spec:
  defaults:
    templates:
      podTemplate: pod-template-with-datadog-annotations
  configuration:
    clusters:
      - name: "clickhouse"
        layout:
          shardsCount: 1
          replicasCount: 2
  templates:
    podTemplates:
    - name: pod-template-with-datadog-annotations
      metadata:
       annotations:
         ad.datadoghq.com/clickhouse.check_names: '["clickhouse"]'
         ad.datadoghq.com/clickhouse.init_configs: "[{}]"
         ad.datadoghq.com/clickhouse.instances: |
          [
            {
              "host": "%%host%%",
              "port": "%%port%%",
              "username": "username",
              "password": "password"
            }
          ]
      spec:
       containers:
         - name: clickhouse
           image: yandex/clickhouse-server:21.3

from clickhouse-operator.

Can you please elaborate your proposition?

from clickhouse-operator.

What is the expected sequence of actions around these annotations?

from clickhouse-operator.

prometheus-operator is able to automatically discover pods to scrape metrics based on SD.
At least the one installed with helm chart https://github.com/helm/charts/blob/master/stable/prometheus/values.yaml#L1478

So if you add these two lines with annotations to deployment, monitoring will be added automatically without need to add ServiceMonitor

annotations:
        prometheus.io/port: '8888'
        prometheus.io/scrape: 'true'

from clickhouse-operator.

I am working in exposing ClickHouse metrics to DataDog using a similar approach (creating annotations), right now it is not possible to create pod annotations using clickhouse-operator (or at least I was not able to do it). I was thinking if at the same time of implementing that for prometheus, maybe the solution could be general enough to be able to do that for other metric collectors.

Edit: In the case I was exposing the annotations must be set to clickhouse pods instead of the operator ones. So I don't know if it would be better to create a new ticket as it seems is not totally related to this issue

from clickhouse-operator.

@jordivilaseca it is still not clear for me - would you like to add custom annotation to operator's pod (because metrics-exporter lives there, however metrics exporter exports metrics in Prometheus format only so far and I have doubts that is useful to DataDog) OR would you like to add custom annotation to pods where clickhouse runs (and in this case DataDog has to know how to fetch metrics from clickhouse directly). Can you, please clarify a little?

from clickhouse-operator.

@sunsingerus Thanks for the replay and sorry for the confusion. I originally wrote here because I thought it was the same case but I found out afterward that were different things.

Datadog agent recently implemented an integration to export metrics (and logs) from Clickhouse. Apart from that, they also have what they call Autodiscovery, that is a way to tell the agent what integration use for each pod using its annotations. So the idea would be to implement a way to add custom annotations to pods where clickhouse runs.

from clickhouse-operator.

Yeah, I mixed them up a little, you're right.
Thanks)

from clickhouse-operator.

@sunsingerus I added custom annotations to my ClickHouseInstallation spec like below, applied it to my clickhouse namespace, and then brought up a new pod by killing the earlier one. The new pod didn't have these custom annotations though. Is this the correct way to go about it? Are there other ways this can be done?

apiVersion: "clickhouse.altinity.com/v1"
kind: "ClickHouseInstallation"
metadata:
  name: "clickhouse"
  annotations:
    ad.datadoghq.com/clickhouse.check_names: '["clickhouse"]'
    ad.datadoghq.com/clickhouse.init_configs: "[{}]"
    ad.datadoghq.com/clickhouse.instances: |
      [
        {
          "host": "%%host%%",
          "port": "%%port%%",
          "username": "username",
          "password": "password"
        }
      ]

from clickhouse-operator.

@Slach That worked! Thank you!

from clickhouse-operator.

@vinayak-mehta @Slach are you sure that these annotations are not supposed to be applied in the datadog helm chart?
according to https://docs.datadoghq.com/agent/kubernetes/integrations/?tab=helm#configuration
and https://docs.datadoghq.com/integrations/clickhouse/?tab=containerized

from clickhouse-operator.

@vinayak-mehta could you share your datadog config which you applied to DataDog agent to integration with kubernetes?

from clickhouse-operator.