Custom log-based metric not recognized by HPA about k8s-stackdriver HOT 6 OPEN

Could you share your HPA configurations after removing sensitive information? cc @az-software-engineer

An example: https://stackoverflow.com/questions/78130483/horizontal-pod-autoscaler-with-stackdriver-custom-metric-in-gke-fails-with-inva

from k8s-stackdriver.

@CatherineF-dev - Please see below.

Custom Metric:

Custom Log-Based Metric:

Example Values:
minReplicas: 1
maxReplicas: 10
averageValue: 50
backend_target_name: xyz

** Is HPA able to scale off Custom Log-Based Metrics (logging.googleapis.com) VS. Custom Metrics (custom.googleapis.com)?

from k8s-stackdriver.

This is a working example https://stackoverflow.com/questions/78130483/horizontal-pod-autoscaler-with-stackdriver-custom-metric-in-gke-fails-with-inva.

Could you follow this to use loadbalacing.googleapis.com|https|request_count?

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: my-autoscale
  namespace: somenamespace
spec:
  maxReplicas: 3
  metrics:
  - pods:
      metric:
        name: prometheus.googleapis.com|jvm_memory_bytes_used|gauge
        selector:
           matchLabels:
             metric.labels.area: heap
      target:
        averageValue: 2G
        type: AverageValue
  type: Pods
  minReplicas: 1
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-java-app

from k8s-stackdriver.

Thank you for the example. We've attempted the provided but using logging.googleapis.com rather than Prometheus. GKE accepts the new config but HPA still remains "Unable to read all metrics". Maybe the example worked due to the Prometheus use case?

from k8s-stackdriver.

For logging.googleapis.com metrics, found a similar example here https://cloud.google.com/kubernetes-engine/docs/tutorials/autoscaling-metrics#pubsub_8

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: pubsub
spec:
  minReplicas: 1
  maxReplicas: 5
  metrics:
  - external:
      metric:
       name: pubsub.googleapis.com|subscription|num_undelivered_messages
       selector:
         matchLabels:
           resource.labels.subscription_id: echo-read
      target:
        type: AverageValue
        averageValue: 2
    type: External
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: pubsub

from k8s-stackdriver.

Unfortunately, that is not working either. In the example provided, you're referencing "pubsub.googleapis.com" which we have successfully configured and tested. However, we're trying to get "logging.googleapis.com" to work using a user-defined custom metric that captures log events.

apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
name: abc-hpa
namespace: abc-frontend
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: abc-server-deployment
minReplicas: {{ .Values.abc.hpa.min_replicas }}
maxReplicas: {{ .Values.abc.hpa.max_replicas }}
metrics:

• external:
  metric:
  name: logging.googleapis.com|user|UserDefinedCustomMetric
  target:
  type: AverageValue
  averageValue: {{ .Values.abc.hpa.averageValue }}
  type: External
  scaleTargetRef:
  apiVersion: apps/v1
  kind: Deployment
  name: abc-server-deployment

from k8s-stackdriver.