Investigate which CKF charms provide metrics by default about bundle-kubeflow HOT 5 CLOSED

Thank you for reporting us your feedback!

The internal ticket has been created: https://warthogs.atlassian.net/browse/KF-5487.

This message was autogenerated

from bundle-kubeflow.

admission-webhook

Code doesn't implement any metrics although there are some references in its go.mod and .sum files.

argo-controller

• Upstream docs on metrics exposed https://argo-workflows.readthedocs.io/en/latest/metrics
• argo-controller.txt

dex

• Code in upstream implementing metrics https://github.com/dexidp/dex/blob/088339fc287a24de23ffcfe7985287cef2a3b2fa/cmd/dex/serve.go#L26-L30
• dex-auth.txt

envoy

• Metrics include (they call them statistics) https://www.envoyproxy.io/docs/envoy/v1.15.0/configuration/upstream/cluster_manager/cluster_stats https://www.envoyproxy.io/docs/envoy/v1.15.0/configuration/http/http_conn_man/stats
• envoy.txt

istio-gateway

• charm endpoint: https://github.com/canonical/istio-operators/blob/e0d2c0261a3760366b87e7be27100393009967ac/charms/istio-gateway/src/manifest.yaml#L39-L40
• Upstream docs on metrics (it's an envoy): https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/observability/statistics.html https://istio.io/latest/docs/ops/configuration/telemetry/envoy-stats/
• istio-gateway.txt

istio-pilot

• upstream docs https://istio.io/latest/docs/concepts/observability/
• istio-pilot.txt

jupyter-controller

• Provides default golang/prometheus and custom metrics
• jupyter-controller.txt

jupyter-ui

Code doesn't implement any metrics.

katib-controller

• Provides default golang/prometheus and custom metrics.
• katib-controller.txt

katib-db-manager

Code doesn't implement any prometheus metrics.

katib-ui

Code doesn't implement any metrics.

kfp-api

• Upstream endpoint https://github.com/kubeflow/pipelines/blob/5399585b6a0f92446bcfc5a7588f2a85ea0fe6a3/backend/src/apiserver/main.go#L208
• custom metrics in code
  • resource-manager
  • experiment-server
  • job-server
  • pipeline-server
  • pipeline-upload
  • run-server
• kfp-api.txt

Pipeline steps don't expose metrics by default. Feature requests:

• kubeflow/pipelines#7341
• kubeflow/pipelines#6336

kfp-metadata-writer

Code doesn't implement anything related to metrics https://github.com/kubeflow/pipelines/tree/master/backend/metadata_writer

kfp-persistence

Code doesn't implement any metrics. The only reference to metrics is about "metrics" provided from the application for exposing artifacts in the UI.

kfp-profile-controller

Code doesn't implement any metrics.

kfp-schedwf

Code doesn't implement any prometheus metrics.

kfp-ui

Code doesn't implement any prometheus metrics.

kfp-viewer

Code doesn't implement any prometheus metrics.

kfp-viz

Code doesn't implement any prometheus metrics.

knative-eventing & knative-serving

• They do expose metrics
  • knative-serving metrics
  • knative-eventing metrics
• However, through one shared exporter (can't use for querying up metric of those charms)
• knative-eventing-serving.txt

knative-operator

• Exposes metrics through this endpoint (but charm doesn't configure its service right now)
• knative-operator.txt

kserve-controller

• Workload exposes metrics at endpoint (uses /metrics by default) although our charm doesn't patch the service to expose this endpoint.
• Implements some kind of default k8s metrics.
• kserve-controller.txt
  Side note that if we 'd like ISVCs to expose metrics too (via their own endpoints - docs), we should modify the values in the configmap according to this guide.

kubeflow-dashboard

• implements metrics but are only accessible in the case of a deployment in GCE. When deploying CKF, that's what the dashboard logs:

   2024-04-02T08:03:21.120Z [serve] > [email protected] serve
   2024-04-02T08:03:21.120Z [serve] > node dist/server.js
   2024-04-02T08:03:21.120Z [serve] 
   2024-04-02T08:03:23.555Z [serve] Initializing Kubernetes configuration
   2024-04-02T08:03:23.611Z [serve] Unable to fetch Application information: 404 page not found
   2024-04-02T08:03:23.611Z [serve] 
   2024-04-02T08:03:23.637Z [serve] "other" is not a supported platform for Metrics
   2024-04-02T08:03:23.638Z [serve] Using Profiles service at http://kubeflow-profiles.kubeflow:8081/kfam
   2024-04-02T08:03:23.645Z [serve] Server listening on port http://localhost:8082 (in production mode)
  

• Curling the pod's endpoint <pod-ip>:80802/api/metrics returns

   {"error":"Operation not supported"}%
  

Thus, metrics are not available from upstream in our case.

kubeflow-profiles

kfam

• implements metrics at endpoint
• golang/prometheus and custom metrics
• kfam.txt

profiles

• implements metrics at endpoint
• golang/prometheus and custom metrics
• kubeflow-profiles.txt

kubeflow-roles

There isn't an upstream app for this charm.

kubeflow-volumes

Code doesn't implement any metrics.

metacontroller

• Implements golang/prometheus plus some custom metrics at endpoint (already exposed by charm)
• metacontroller-operator.txt

minio

• Upstream docs on metrics exposed at endpoints with some info
• minio.txt

mlmd

Code doesn't implement any metrics.

oidc-gatekeeper

There are some references to prometheus packages in go.mod and .sum files but nothing is implemented in its code.

pvcviewer-operator

• Code defines a metric endpoint and also has the corresponding manifests too (auth endpoint, serviceMonitor)
• However, I couldn't access those although the charm exposes the port

   k exec envoy-operator-0 -n kubeflow -- curl pvcviewer-operator.kubeflow.svc.cluster.local:8443/metrics
   Client sent an HTTP request to an HTTPS server.
  

  This is probably due to this note in upstream manifests. We should probably remove this from our charm command (although I didn't try and need to verify this). It could also be that we have not included the Prometheus manifests in our case.

• Thus, no text file containing exposed metrics here.

seldon-controller-manager

• Implements metrics.
• Upstream docs on metrics and here
• seldon-core.txt

tensorboard-controller

• Implements metrics and defines an endpoint in manifests. This also (like pvcviewer-operator) has references about putting metrics behind auth so we should be careful when implementing it (looks like now it's not behind auth)
• tensorboard-controller.txt

tensorboard-web-app

Code doesn't implement any metrics.

training-operator

• Metrics are exposed in endpoint
• Golang/prometheus and custom metrics are implemented. Custom metrics are also described here.
• (not 100% releveant but) Jobs implement some custom metrics too (references 1, 2 and 3)
• training-operator.txt

from bundle-kubeflow.

Upstream apps that do not already expose metrics (wip)

1. admission-webhook
2. jupyter-ui (jupyter-web-app)
3. katib-db-manager
4. katib-ui
5. kfp-metadata-writer
6. kfp-persistence
7. kfp-profile-controller
8. kfp-schedwf
9. kfp-ui
10. kfp-viewer
11. kfp-viz
12. kubeflow-dashboard (explanation in previous comment)
13. kubeflow-roles. This isn't an upstream app but we 'd still need an exporter if we 'd like metrics from this charm.
14. kubeflow-volumes (volumes-web-app)
15. mlmd
16. oidc-authservice
17. tensorboards-web-app

from bundle-kubeflow.

Regarding all the K8s Controllers from kubeflow/kubeflow (notebooks, profiles, tensorboards) they will get some quite useful metrics by default because of controller-runtime golang package, that comes with Kubebuilder
https://book.kubebuilder.io/reference/metrics-reference

Those are perfect for capturing if the controllers are working as expected, and it's great it will be handled by default.

In order for this to happen though, someone upstream will need to bump the controller-runtime package from 0.11 to 0.16.3

from bundle-kubeflow.

Related sheet

from bundle-kubeflow.