Queues scale temporarily for to -1 after autoscaler restart about k8s-worker-pod-autoscaler HOT 6 CLOSED

I just took that for a spin and it seems to solve the issue!

❯ ksys logs -f deploy/workerpodautoscaler | grep dev-inventa-root-user-queue
W0921 18:11:57.103341       1 controller.go:430] dev-queue qMsgs: -1 not uninitialized, waiting for init to complete
I0921 18:12:15.097447       1 controller.go:450] dev-queue qMsgs: 6869, desired: 4

I'll keep an eye out for when you cut a new version, but this is working great in the meantime.

from k8s-worker-pod-autoscaler.

It might be a bug. Thanks for reporting.

What do you mean by WPA to -1 here? Does it mean desired replica is being set to -1?

it temporarily sets the value on all of my wpas to -1,

Many values are initialised to -1 on startup. They are updated as we get their recent values. There could be a bug causing this, need to debug. Detailed logs of the event can help me troubleshoot.

from k8s-worker-pod-autoscaler.

I'll try to demo what I'm seeing with some more concrete examples.

But essentially, we run most of our workloads on AWS Spot instances, so then can go away/restart at any time (I've also seen a random error caused the pod to crash or restart).

I can easily see this if i restart the pod manually:

kubectl -n kube-system rollout restart deploy/workerpodautoscaler

So when that happens what I see:

k get wpa -ocustom-columns=NAME:.metadata.name,QUEUE:.status.CurrentMessages,DEPLOYMENTS:.status.AvailableReplicas,MIN:.spec.minReplicas,MAX:.spec.maxReplicas <redacted-wpa-name> -w
NAME                  QUEUE   DEPLOYMENTS   MIN   MAX
<redacted-wpa-name>   3       1             0     4
<redacted-wpa-name>   -1      1             0     4
<redacted-wpa-name>   2       0             0     4
<redacted-wpa-name>   0       1             0     4

and for the deployment

❯ k get deploy <redacted-deployment> -w
NAME                    READY   UP-TO-DATE   AVAILABLE   AGE
<redacted-deployment>   1/1     1            1           7d
<redacted-deployment>   1/0     1            1           7d
<redacted-deployment>   1/0     1            1           7d
<redacted-deployment>   0/0     0            0           7d
<redacted-deployment>   0/1     0            0           7d
<redacted-deployment>   0/1     0            0           7d
<redacted-deployment>   0/1     0            0           7d
<redacted-deployment>   0/1     1            0           7d
<redacted-deployment>   1/1     1            1           7d

So essentially the queues gets set to -1, the deployment gets scaled to zero, all the pods terminate, it gets the queue value from sqs, and then scales the deployment back up to 1, new pods restart.

Also, as I mentioned, I have a LOT of wpa and deployments (in the hundreds) so this causes a huge swing of pod restrats across the entire cluster.

from k8s-worker-pod-autoscaler.

I would think, upon startup, that if a value currently exists, it wouldn't reset it back to -1 unless it was unsuccessfully able to get metrics from SQS, and then maybe then.

from k8s-worker-pod-autoscaler.

Thanks for the details.

-1 in queue size should not cause scale down. As it is uninitialized value.

Let me find the issue and fix the bug.

from k8s-worker-pod-autoscaler.

@aleclerc-sonrai Trying out a fix in https://github.com/practo/k8s-worker-pod-autoscaler/pull/137/files

Would you be able to use this image for your WPA pod and see if this fixes the issue?

practodev/workerpodautoscaler:v1.4.1-1-g73ba5c4

from k8s-worker-pod-autoscaler.