Unable to fetch queue messages about k8s-worker-pod-autoscaler HOT 8 OPEN

I generated temporary credentials manually using the AssumeRole. I believe it is working now.

Previously my node's role permissions included the following as per this policy:

"cloudwatch:GetMetricData"
"sqs:GetQueueAttributes"
"sqs:ReceiveMessage"

It was resolved by granting all read permissions on SQS:

"cloudwatch:GetMetricData"
"sqs:GetQueueAttributes"
"sqs:GetQueueUrl"
"sqs:ListDeadLetterSourceQueues"
"sqs:ListQueueTags"
"sqs:ListQueues"
"sqs:ReceiveMessage"

My restarted WPA no longer logs any errors.

from k8s-worker-pod-autoscaler.

Possible to share the complete log?

from k8s-worker-pod-autoscaler.

Hi @alok87 : please see the example included in my issue. The WPA starts spitting out Unable to fetch no of messages messages as soon as the container starts. There are no other kinds of log messages.

Note that I have anonymized the account number and queue name.

E1104 12:42:26.926463       1 sqs.go:406] Unable to fetch no of messages to the queue "queue", Client not found for queue: https://sqs.us-east-1.amazonaws.com/<account>/queue
.
E1104 12:42:26.926476       1 sqs.go:406] Unable to fetch no of messages to the queue "queue", Client not found for queue: https://sqs.us-east-1.amazonaws.com/<account>/queue
.
E1104 12:42:26.926498       1 sqs.go:406] Unable to fetch no of messages to the queue "queue", Client not found for queue: https://sqs.us-east-1.amazonaws.com/<account>/queue
.
E1104 12:42:26.926513       1 sqs.go:406] Unable to fetch no of messages to the queue "queue", Client not found for queue: https://sqs.us-east-1.amazonaws.com/<account>/queue
.
E1104 12:42:26.926527       1 sqs.go:406] Unable to fetch no of messages to the queue "queue", Client not found for queue: https://sqs.us-east-1.amazonaws.com/<account>/queue
.
E1104 12:42:26.926540       1 sqs.go:406] Unable to fetch no of messages to the queue "queue", Client not found for queue: https://sqs.us-east-1.amazonaws.com/<account>/queue
.
E1104 12:42:26.926552       1 sqs.go:406] Unable to fetch no of messages to the queue "queue", Client not found for queue: https://sqs.us-east-1.amazonaws.com/<account>/queue

from k8s-worker-pod-autoscaler.

Does the queue exist in sqs? Possible to try using sqs client with same creds and see data comes?

Just want to rule out the possibility of configuration issue first

from k8s-worker-pod-autoscaler.

can we close this?

from k8s-worker-pod-autoscaler.

Do you think we should update something in the doc here on policy, https://github.com/practo/k8s-worker-pod-autoscaler#install

from k8s-worker-pod-autoscaler.

I feel like there may be something else missing.

Even though I get no permissions errors, I am unable to trigger a scaling operation on the deployment. Any ideas?

I have 10000+ messages in the queue and only one deployment pod running.

k get pods
NAME                                 READY   STATUS    RESTARTS   AGE
example-deployment-795d868d4-8nzfv   1/1     Running   0          7m19s

Does the WPA require some kind of write or tag attributes?

I can submit a PR for the documentation once I confirm this is working.

from k8s-worker-pod-autoscaler.

WPA has verbosity in logs, may be try that. -v=4

• Also share the output of WPA yaml

k get wpa -o yaml <wpa_object>

• check if deployment replicas changed with queue length
• check the queue length in AWS shows the 1000 messages? sqs metrics picture if posted here can help.

from k8s-worker-pod-autoscaler.