Client blocks indefinitely on high volume about aws-sdk-kotlin HOT 4 CLOSED

Hi @steamstreetjon, thank you for the bug report. Without more information it's a little difficult to know how to proceed. Could you please provide a bit more detail to help us diagnose the issue?

• How do you know the SDK is blocking? Are you seeing exceptions/timeouts?
• You mentioned ~500k/hr requests—how is that split across your function instances? Is each request a separate instance or does a single instance share multiple simultaneous requests?
• How are you wrapping the call to the SDK's suspend operation to get a coroutine context?
• Can you share any relevant code?
• Have you examined the logs at info/debug/trace levels for pertinent messages? Can you share any relevant log lines?

from aws-sdk-kotlin.

I'll answer a few of this, but I'd like some help in understanding the best way to give you what you need.

The 500k is a single lambda that is provisioned as normal. There are certainly spikes in activity, but the lambdas are pretty standard, triggered by EventBridge events.

The initial tell was that the Lambda was timing out. As I dug in further I could see that it was happening at a specific point. I see the log for the request, but then nothing until the Lambda times out. Since it's event bridge, the event is retried shortly after and the exact same details work fine.

Here's a screenshot of the logs. I can share more logging details, just didn't want to be too public about it, but you can see it gets to the GetItem POST and then hangs until the lambda is complete.

For reference, when it runs the second time a minute later, the 'healthy' version gets a response and carries on:

I tried to wrap the code in a withTimeout coroutine, and that is not firing, which makes me thing it's legit blocked... like there's some synchronization deadlock in multi-threaded code (otherwise the coroutine should be able to handle the timeout).

I'd really like to help track this down with you, so let me know if you'd like to do some debugging or if there is some logging to enable to help you understand it better.

Also, here's the configuration of the client right now (I added the timeouts to see if I could get the client to fail):

            DynamoDbClient.fromEnvironment {
                logMode = LogMode.LogRequestWithBody + LogMode.LogResponseWithBody
                httpClient {
                    socketReadTimeout = 5.seconds
                    connectTimeout = 5.seconds
                }
            }

from aws-sdk-kotlin.

Thanks for the additional info. It's particularly curious to me that withTimeout around the DDB call isn't having an effect. As you mention, that may be an indication of a hard block somewhere. Interestingly, though, you mentioned this happens with both OkHttp and CRT engines, both of which use wildly different parallelism models. That points to a potential problem outside of the HTTP engines.

Unfortunately, there's still not enough info present to assist in root causing. We have various tests of the SDK in our release pipeline, including some which test high concurrency and high throughput, which have not exhibited this issue before. To dive deeper on this, would you be willing to provide repro code and full trace-level logs for a succeeding and failing request?

If you have an AWS Support plan, you can submit a support case through the AWS Console with your logs/code which will make it available for us internally. If not, you can attach or paste your logs/code in this issue after redacting sensitive info like access keys, internal endpoints, etc.

from aws-sdk-kotlin.

It looks like this issue has not been active for more than 5 days. In the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please add a comment to prevent automatic closure, or if the issue is already closed please feel free to reopen it.

from aws-sdk-kotlin.