Throttling: Rate exceeded, status code: 400 about cloudquery HOT 7 CLOSED

@yevgenypats Thanks a lot for the response and the update. I've added both log_level: debug and max_retries: 20 and I can now fetch all iam data from all our accounts without any errors! I assume this is due to the max_retries configuration, thanks again.

from cloudquery.

Typically, AWS will work with you to raise the soft limits on your accounts. That's likely the best way to tackle this issue if you need to query all of the accounts at once on a regular basis. I don't believe anything could be done on the cloudquery side other than doing client side throttling.

from cloudquery.

Thanks @mvaerle and @sethcenterbar I believe raising the AWS soft limit might work indeed.
I can also try and add a configurable option to increase the number of retries done by AWS SDK.
Also, I'll add a logging option so you can rerun it and we can see on which retry you get the error.
As a last resort I guess we will need to implement client side throttling.

from cloudquery.

Thanks for the quick reply!

Asking AWS to raise limits might work, but I'm not even sure which limits I'm hitting and by how much. We're never running into these except when using cloudquery. I'm trying to get some insight through cloudtrail but no luck so far, I'll report back if I can find something.

Once there is logging available I'm happy to test and try again. For now I can use the workaround of commenting out accounts and/or contact AWS to raise our limits.

from cloudquery.

@mvaerle I've added log_level: debug and max_retries: 20 to your AWS config.yml as specified here.

This should show us both what limit is being reached and also if changing max_retries to a higher number helps or not?

Those keywords are available at v0.5.3

from cloudquery.

1. It's good to gather data on your API usage which can be done with cloudtrail and EK

2. If you're using the AWS SDK I would assume it's handling the retry strategy and that may be configurable and worth checking out for short-term relief. In some cases caching can be enabled. If not and you're using it often you might add a caching layer or set up a caching aws proxy. If you're rolling your own retries, then use the AWS blog post on retries to implement a good backoff strategy (exponential + jitter).

3. If 2 finally fails you, hit up AWS and they will raise the limit. Not all limits can be raised.

from cloudquery.

Added max_retries + max_backoff to work with aws sdk v2. Available >0.2.3

PR: cloudquery/cq-provider-aws#10 By default its 5 retries and 30 seconds exponential backoff

from cloudquery.