Comments (3)
While I can see that in my GKE Autopilot clusters the event-exporter-gke
pods are running in the kube-system
namespace, there does not appear to be a pre-installed custom-metrics-stackdriver-adapter.
When installing in GKE Autopilot, you do not need need to, and in fact you cannot use node iam.gke.io/gke-metadata-server-enabled: "true"
. Otherwise the directions for how to Use Workload Identity are correct
See #315 and GoogleCloudPlatform/kubernetes-engine-samples#140
gcloud iam service-accounts create custom-metrics-sd-adapter --project "$GCP_PROJECT_ID"
gcloud projects add-iam-policy-binding "$GCP_PROJECT_ID" \
--member "serviceAccount:custom-metrics-sd-adapter@$GCP_PROJECT_ID.iam.gserviceaccount.com" \
--role "roles/monitoring.editor"
gcloud iam service-accounts add-iam-policy-binding \
--role roles/iam.workloadIdentityUser \
--member "serviceAccount:$GCP_PROJECT_ID.svc.id.goog[custom-metrics/custom-metrics-stackdriver-adapter]" \
"custom-metrics-sd-adapter@$GCP_PROJECT_ID.iam.gserviceaccount.com"
kubectl create -f https://raw.githubusercontent.com/GoogleCloudPlatform/k8s-stackdriver/master/custom-metrics-stackdriver-adapter/deploy/production/adapter.yaml
kubectl annotate serviceaccount custom-metrics-stackdriver-adapter \
"iam.gke.io/gcp-service-account=custom-metrics-sd-adapter@$GCP_PROJECT_ID.iam.gserviceaccount.com" \
--namespace custom-metrics
from k8s-stackdriver.
same problem.
clearly Autopilot cluster supports the external metrics from Cloud Monitoring -- it even has UI dialog to set them.
Though when you try to create a HPA object using the official example
https://cloud.google.com/kubernetes-engine/docs/tutorials/autoscaling-metrics#pubsub
it fails:
$ kubectl describe horizontalpodautoscalers.autoscaling
Name: myservice
Namespace: default
Labels: <none>
Annotations: <none>
CreationTimestamp: Mon, 12 Jun 2023 18:50:12 +0300
Reference: Deployment/myservice
Metrics: ( current / target )
"pubsub.googleapis.com|subscription|num_undelivered_messages" (target average value): <unknown> / 2
Min replicas: 1
Max replicas: 5
Deployment pods: 1 current / 1 desired
Conditions:
Type Status Reason Message
---- ------ ------ -------
AbleToScale True ReadyForNewScale recommended size matches current size
ScalingActive False FailedGetExternalMetric the HPA was unable to compute the replica count: unable to get external metric default/pubsub.googleapis.com|subscription|num_undelivered_messages/nil: unable to fetch metrics from external metrics API: the server could not find the requested resource (get pubsub.googleapis.com|subscription|num_undelivered_messages.external.metrics.k8s.io)
ScalingLimited False DesiredWithinRange the desired count is within the acceptable range
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedGetExternalMetric 1s horizontal-pod-autoscaler unable to get external metric default/pubsub.googleapis.com|subscription|num_undelivered_messages/nil: unable to fetch metrics from external metrics API: the server could not find the requested resource (get pubsub.googleapis.com|subscription|num_undelivered_messages.external.metrics.k8s.io)
I'm pretty sure that the metric exists -- I can see it in Cloud Monitoring
from k8s-stackdriver.
ok. it works after installing custom metric adapter. I thought it should be preinstalled
kubectl apply -f https://raw.githubusercontent.com/GoogleCloudPlatform/k8s-stackdriver/master/custom-metrics-stackdriver-adapter/deploy/production/adapter_new_resource_model.yaml
from k8s-stackdriver.
Related Issues (20)
- custom-metrics-sd-adapter: 403s when fetching metrics from another project
- Autoscaler throws FailedGetExternalMetric after upgrade to 1.22 HOT 1
- HPA doesn't scale down to minimum size when no data is available from external metrics API server
- prometheus-to-sd | 403 Permission error, with correct Service Account permissions. HOT 2
- Custom stack driver pod doesnt seem to have liveness/readiness probe.
- Running custom-metrics-stackdriver-adapter as non-root user HOT 1
- Stackdriver metrics are not supported by HorizontalPodAutoscaler HOT 2
- ClusterRole in stackdriver adapter has namespace defined
- custom-metrics-stackdriver-adapter not working - auth problems? HOT 13
- Security Policy violation Binary Artifacts HOT 5
- Allow custom metrics from a different pod HOT 3
- Custom metrics adapter spewing errors "apiserver was unable to write a fallback JSON response: http2: stream closed" HOT 11
- custom-metrics-stackdriver-adapter - couldn't get resource list for external.metrics.k8s.io/v1beta1: Got empty response for: external.metrics.k8s.io/v1beta1 HOT 3
- istio.io/service/server/response_latencies metric that HPA collected is different to Cloud monitoring HOT 1
- Filtering metrics by labelSelector in external.metrics.k8s.io api doesn't work HOT 1
- Timeout error logs HOT 5
- Deploying adapter_new_resource_model.yaml results in OOMKilled HOT 3
- 100% memory and CPU and never recovers HOT 2
- Documentation bugs in custom-metrics-stackdriver-adapter README HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from k8s-stackdriver.