WPA controller was unable to update the number of replicas: status.currentMetrics in body must be of type array: "null" about watermarkpodautoscaler HOT 4 CLOSED

Hello @deiga !
Thank you for this report, I have seen this in the past and the issue was that the external metrics provider (Datadog Cluster Agent in my case) was not exposing the metric during the first reconcile loop. So the controller could not write the status and was surfacing this error.

Could you confirm that the metric is available on the external metrics route ?
Something like:
kubectl get --raw "/apis/external.metrics.k8s.io/v1beta1/namespaces/web-services-staging/php_fpm.listen_queue.size"
or if you have multiple, I think this would work:
kubectl get --raw "/apis/external.metrics.k8s.io/v1beta1/namespaces/web-services-staging/php_fpm.listen_queue.size?labelSelector=app%3Dnextapi,stage%3Dstaging,region%3Deu-west-1"

I am going to share with my team though to investigate and better handle the error.

from watermarkpodautoscaler.

Thanks for your quick reply @CharlyF

Neither of those queries works, they return Error from server (ServiceUnavailable): the server is currently unable to handle the request

I'm beginning to suspect that there is an issue with our Cluster Agent. Although the cluster agent reports the External metrics as valid

from watermarkpodautoscaler.

Ah, indeed that explains it then. If these queries do not return, could you verify that the APIService for external metrics is correctly setup as well as the service itself has the cluster agent registered as endpoints ?
For standard deployments that would be
kubectl describe apiservice v1beta1.external.metrics.k8s.io
kubectl describe service datadog-cluster-agent

from watermarkpodautoscaler.

We improved the logging in the recent minor release of the controller but I believe this issue was with the Cluster Agent and not the WPA.
I am going to close this for now, but feel free to reopen if this is still an issue for you.

from watermarkpodautoscaler.