elastic / elasticsearch-k8s-metrics-adapter Goto Github PK

TLS cert validation is skipped for the moment. Use should be able to configure TLS settings for both the upstream and Elasticsearch server.

Metrics list is only retrieved once, during startup.

The metric adapter should reload the list of available metrics list periodically from the metrics servers.

In a first implementation we could terminate the adapter if an error occurs while fetching this list from one of the upstream server.

Since a same metric can be exposed by more than one metric server, it would be useful to expose a debug API to let the user understand which metric server is used to serve a given metric.

This new API should be protected by an authentication mechanism and support TLS, like the other ones used to serve the custom metrics API.

GET /apis/explain/container_cpu_cfs_periods_total

{
	"name": "container_cpu_cfs_periods_total",
	"metricSources": [{
		...
	}, {
		...
	}]
}

Besides the configuration related to the Pod itself (ServiceAccount to be used, permissions, resources to be allocated) 2 important items are:

1. the adapter configuration file (which is stored in a ConfigMap)
2. the secret which contains the credentials used to connect to Elasticsearch

see this example if you are not familiar with the configuration file

We could:

• Provide some samples in a deploy/manifests/ directory, as it is the case for the Prometheus adapter. One downside is that the user would have to:

  • Generate the Secret with the credentials manually
  • Edit/customize the ConfigMap to add an existing metrics server.

• Provide an Helm chart. This is also an option in the context of the Prometheus adapter.

• Provide a script or add a target to the Makefile to generate a quickstart manifest.

Or at least I assume that in https://github.com/elastic/elasticsearch-k8s-metrics-adapter/blob/main/README.md?plain=1#L101 the link elastic/cloud-on-k8s@master...barkbay:autoscaling/kibana-poc is something that's already merged back now and should point to the ECK repo?

Within an HPA spec. a label selector can be used to describe a metric:

Many metrics pipelines allow you to describe metrics either by name or by a set of additional descriptors called labels. For all non-resource metric types (pod, object, and external, described below), you can specify an additional label selector which is passed to your metric pipeline. For instance, if you collect a metric http_requests with the verb label, you can specify the following metric block to scale only on GET requests:

type: Object
object:
  metric:
    name: http_requests
    selector: {matchLabels: {verb: GET}}

This selector uses the same syntax as the full Kubernetes label selectors. The monitoring pipeline determines how to collapse multiple series into a single value, if the name and selector match multiple series. The selector is additive, and cannot select metrics that describe objects that are not the target object (the target pods in the case of the Pods type, and the described object in the case of the Object type).

(see original documentation here)

For the moment metricSelector labels.Selector is not used to retrieve a metric:

If we want to support this feature we need to understand what is a label in the context of a document stored in Elasticsearch.

Note that the same question can be raised in the context of External metrics, however they are not likely to be supported as part of a first release of the adapter:

Instead of having a full Elasticsearch client configuration (url, username, password, ca), we could let the user set a reference to an Elasticsearch cluster managed by ECK.
It would allow the metrics adapter to setup automatically the URL and the certificates. We could use the elastic user in a first step, I don't think it would be easy to automatically setup a dedicated Elasticsearch user with the right set of privileges.
One downside of this feature is that it would require the metrics adapter to read Secrets which are likely to exist in a different namespace.

There is an error with this repository's Renovate configuration that needs to be fixed. As a precaution, Renovate will stop PRs until it is resolved.

Error type: Cannot find preset's package (local>elastic/renovate-config:control-plane-serverless)

• Which namespace should be used:

  • Should we reuse the eck namespace ? docker.elastic.co/eck/elasticsearch-adapter:latest
  • Should we use a dedicated namespace ? docker.elastic.co/elasticsearch-adapter/elasticsearch-adapter:latest

• It also raises the question of the version for the first release ? 0.8, 0.9, 1.0, alpha1 ?

I'm not sure it's worth investing some time in a CI pipeline for now. For the first release I think it should be fine to manually:

1. create a tag in the git repository
2. create the Docker image and push it into the registry using a target in the Makefile for example.

Configuration is loaded from a configuration file, stored in a ConfigMap. The adapter should automatically stop if there's a change. Users should also be able to opt out if they want to manage a rolling restart themselves.

We should generate a notice file using https://github.com/elastic/go-licence-detector, and ideally maintain it as part of the pr process.

It should be possible to change the name of the metrics as they are exposed to Kubernetes. It could be useful when some metrics are available on different servers but share the same name. By using the name, the user is then able to select which metrics server should be used.

For example:

metricServers:
  - name: server1
    type: elasticsearch
    rename:
      matches: "^(.*)$"
      as: "${1}@server1"

All the metrics on server1 can be accessed with the postfix @server1

Inspired from https://github.com/kubernetes-sigs/prometheus-adapter/blob/master/docs/config.md#naming

We should at least have a README.md and a documentation to describe the install process (see #13)

The initial documentation should also offer a "Configuration Walkthroughs" or "Configuration Reference" similar to what exists for Prometheus.

Right now, if a metric is missing in Elasticsearch, we return an error: custom metric <xxx> is not served by any metric client.
This is causing issues in environments with little traffic, where the metric could be missing just because there hasn't been any requests to generate it.

By making it an error, we prevent HPAs from fully synchronizing with no errors.
We should instead return 0, to avoid that error.
We may also want to log a debug/warning when this happens, just to ensure visibility.

cc @barkbay

For now metrics are only associated with Pod resources:

{
	"query": {
		"bool": {
			"must": [{
				"exists": {
					"field": "%s"
				}
			}, {
				"match": {
					"kubernetes.namespace": "%s" -- here
				}
			}, {
				"match": {
					"kubernetes.pod.name": "%s" -- here
				}
			}]
		}
	}
}

Not all metrics make sense in the context of Pods. For example it does not make sense to expose system.memory.free as it is not possible to associate this metric to a specific Pod. The first approach so far has been to filter on the metric name and assume documents also have the kubernetes.namespace and kubernetes.pod.name fields:

metricSets:
  - indices: [ 'metricbeat-*' ]
    fields:
      - patterns: [ '^prometheus\.metrics\.' , '^kibana\.stats\.'  ] # only serve metrics with namespace and pod name

We then assume that we get that kind of document from Elasticsearch:

{
  "_index": "metricbeat-7.14.0-2021.09.21-000001",
  "_type": "_doc",
  "_id": "10V6DXwBJBC0gYXF1JYC",
  "fields": {
     "prometheus.metrics.go_memstats_frees_total": [
      123418991
    ],
   [ ... other metrics starting with prometheus.metrics* ...]
   "kubernetes.pod.name": [
      "kube-dns-b4f5c58c7-47twh"
    ],
   "kubernetes.namespace": [
      "kube-system"
    ],
    "@timestamp": [
      "2021-09-22T12:28:44.448Z"
    ]
  }
}

We may want to improve this association mechanism, for example by allowing other resources to be associated with a given metric. Users may also want to use other fields than kubernetes.namespace or kubernetes.pod.name to query for a given metric.

See here how associations are managed with the Prometheus adapter.

We should add a check to ensure that the proper license is in use.
The check might be added as a new target in the Makefile and, at least, run before the docker-build target.

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

• chore(deps): update dependency go to v1.22.4
• fix(deps): update kubernetes packages to v0.30.2 (k8s.io/apimachinery, k8s.io/apiserver, k8s.io/client-go, k8s.io/component-base, k8s.io/metrics)
• fix(deps): update module github.com/elastic/go-elasticsearch/v8 to v8.14.0
• fix(deps): update module github.com/kimmachinegun/automemlimit to v0.6.1
• fix(deps): update module github.com/prometheus/client_golang to v1.19.1
• fix(deps): update module k8s.io/klog/v2 to v2.130.0
• fix(deps): update module go.elastic.co/apm to v2
• fix(deps): update module go.elastic.co/apm/module/apmelasticsearch to v2
• Click on this checkbox to rebase all open PRs at once

Detected dependencies

• Check this box to trigger a request for Renovate to run again on this repository