Error while using aws-secretsmanager with aws-java-sdk-sts about aws-secretsmanager-jdbc HOT 16 CLOSED

Maybe I confused you. So, let me explain.

If the sts dependency is not present in the classpath (i.e. not inside build.gradle for my case), it will use the node's IAM role, which is something we do not want, as you also said. When we add the sts dependency, it uses the service account role. In order to achieve this with this specific library, it seems that you need to add the v1 of this sts dependency (i.e. com.amazonaws:aws-java-sdk-sts:1.11.1004).
I am using version 1.0.6 of the aws-secretsmanager-jdbc library

I hope it is more clear now!

from aws-secretsmanager-jdbc.

Thank you for opening this issue - we are looking into it.

from aws-secretsmanager-jdbc.

@ktzevelekidis

Maybe I confused you. So, let me explain.

If the sts dependency is not present in the classpath (i.e. not inside build.gradle for my case), it will use the node's IAM role, which is something we do not want, as you also said. When we add the sts dependency, it uses the service account role. In order to achieve this with this specific library, it seems that you need to add the v1 of this sts dependency (i.e. com.amazonaws:aws-java-sdk-sts:1.11.1004). I am using version 1.0.6 of the aws-secretsmanager-jdbc library

I hope it is more clear now!

This needs to be made clear and more visible. Thanks for the solution.

from aws-secretsmanager-jdbc.

You can't use the V1 version of the AWS SDK STS library ('com.amazonaws`) with version 2 of the JDBC caching library, and vice versa.

It's worth nothing that the Java SDK changed classpaths when moving from V1(com.amazonaws) to V2 (software.amazon.awssdk). However, additional libraries vended by AWS like the crypto SDK and this library did not change their classpaths. This was intentional, but I understand the confusion.

If you want to use the V2 SDK, make sure to add version 2 of this library to your project file, and vice versa.

from aws-secretsmanager-jdbc.

Hello,
I am facing a similar issue. In my case, even when I add the sts module on the classpath (in my application's build.gradle), it still does not use the correct credentials provider (i.e. WebIdentityTokenFileCredentialsProvider). It uses the node's IAM role.

Did you manage to find a solution?

I was thinking to download the library, add the sts dependency in the pom.xml, build it and then use (hoping that something will change).

from aws-secretsmanager-jdbc.

I've checked this library, it uses a pretty old version of aws as of now which doesn't support web identity token so even including sts module won't belp in this case. This needs an update to use compatible aws sdk version. As of now I've not found a solution for this. I'll be mostly using at sealed secrets for the time being. May also look into modifications needed in this code to support sts

from aws-secretsmanager-jdbc.

I added this dependency in my build.gradle:

• com.amazonaws:aws-java-sdk-sts:1.11.1004

and it worked for me. However, I get an exception the first time the application starts:

com.mysql.cj.jdbc.exceptions.CommunicationsException: Communications link failure The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.

As a consequence the application restarts and then it works.

from aws-secretsmanager-jdbc.

So it's using the node's iam role instead of the service account injected into pod right. I didn't want the node to have dB access so decided to dropped it. Was running short of time so had to look into alternative approaches

from aws-secretsmanager-jdbc.

No, it is using the service account role and not the node's role. The only issue is that in the first boot, I get the aforementioned exception, and a restart takes place. After this it works as expected. I will investigate further and I will inform you for my findings. Anyway, it would be good if the library will be migrated to aws sdk v2. I saw that there is an open PR for the migration from November, but not yet reviewed.

from aws-secretsmanager-jdbc.

@ktzevelekidis you mentioned initially that it was picking nodes iam role can you clarify more. For me when I add sts module which supports it, the pod just throws a stack trace. It looked to me like circular dependency error but it never managed to work

from aws-secretsmanager-jdbc.

I'm getting the same error - when I use the DefaultAWSCredentialsProviderChain locally, and I provide only the assumed role via the AWS_ROLE_ARN and AWS_ROLE_SESSION_NAME , I get the error, but when I also give it an access key and secret, it works as expected. I'm testing a deploy to ECS now and getting this error.

from aws-secretsmanager-jdbc.

still not working for service role?

from aws-secretsmanager-jdbc.

We have the same issue on EKS. AWS_ROLE_ARN is set through a service-account. But it still uses the assumed role of the node.

from aws-secretsmanager-jdbc.

For those of you facing issues, could you tell me

• whether your cluster has IAM Roles for Service Accounts set up. The instructions to do that are here
• whether the AWS Java SDK in your dependency closure (mvn dependency:tree) is newer than 1.11.704, in order to support IRSA

We're going to document this to make it clearer how to ensure you are using the correct role for calls to Secrets Manager in this library. The behavior described in the issue seems consistent with other AWS APIs and libraries. Please let us know if you find discrepancies that suggest otherwise.

from aws-secretsmanager-jdbc.

Hey @simonmarty,

I'm running into this issue too. Meaning: When I do NOT include sts dependecy, it tries to use the IAM role of the node. When I do include the 2.x version of the sts dependecy I get the following error message:

Caused by: com.amazonaws.SdkClientException: Unable to load AWS credentials from any provider in the chain: [EnvironmentVariableCredentialsProvider: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY)), SystemPropertiesCredentialsProvider: Unable to load AWS credentials from Java system properties (aws.accessKeyId and aws.secretKey), com.amazonaws.auth.profile.ProfileCredentialsProvider@3eec8583: profile file cannot be null, com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper@52657d5f: Unauthorized (Service: null; Status Code: 401; Error Code: null; Request ID: null)]

You mentioned

The behavior described in the issue seems consistent with other AWS APIs and libraries.

However, I do see inconsistent behaviour. As, with the sts library v2 I was able to use the software.amazon.awssdk:* libraries correctly using the EKS service account role and I have used the following imports in gradle for that:

    implementation 'software.amazon.awssdk:secretsmanager:2.21.0'
    implementation 'software.amazon.awssdk:sts:2.21.0'
    implementation 'software.amazon.awssdk:s3:2.21.0'
    implementation 'software.amazon.awssdk:dynamodb:2.21.0'

However, the 'com.amazonaws.secretsmanager:aws-secretsmanager-jdbc:1.0.6' library seems to unfortunately be an exception to this.

from aws-secretsmanager-jdbc.

Hey @simonmarty,

I'm running into this issue too. Meaning: When I do NOT include sts dependecy, it tries to use the IAM role of the node. When I do include the 2.x version of the sts dependecy I get the following error message:

Caused by: com.amazonaws.SdkClientException: Unable to load AWS credentials from any provider in the chain: [EnvironmentVariableCredentialsProvider: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY)), SystemPropertiesCredentialsProvider: Unable to load AWS credentials from Java system properties (aws.accessKeyId and aws.secretKey), com.amazonaws.auth.profile.ProfileCredentialsProvider@3eec8583: profile file cannot be null, com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper@52657d5f: Unauthorized (Service: null; Status Code: 401; Error Code: null; Request ID: null)]

You mentioned

The behavior described in the issue seems consistent with other AWS APIs and libraries.

However, I do see inconsistent behaviour. As, with the sts library v2 I was able to use the software.amazon.awssdk:* libraries correctly using the EKS service account role and I have used the following imports in gradle for that:

    implementation 'software.amazon.awssdk:secretsmanager:2.21.0'
    implementation 'software.amazon.awssdk:sts:2.21.0'
    implementation 'software.amazon.awssdk:s3:2.21.0'
    implementation 'software.amazon.awssdk:dynamodb:2.21.0'

However, the 'com.amazonaws.secretsmanager:aws-secretsmanager-jdbc:1.0.6' library seems to unfortunately be an exception to this.

Apparently I mixed to different version of the sts library. There is software.amazon.awssdk:sts and com.amazonaws:aws-java-sdk-sts. Quite confusing, honestly.

You can see the dependencies I used to get it working for both the (1) software.amazon.awssdk:* dependecies and the (2) com.amazonaws:* dependencies below:

dependencies {
    ...
    implementation 'software.amazon.awssdk:secretsmanager:2.21.0'
    implementation 'software.amazon.awssdk:sts:2.21.0' <------ (1) ---
    implementation 'software.amazon.awssdk:s3:2.21.0'
    implementation 'software.amazon.awssdk:dynamodb:2.21.0'
    ...
    implementation 'com.amazonaws:aws-java-sdk-sts:1.11.1004' <------ (2) ---
    implementation 'com.amazonaws.secretsmanager:aws-secretsmanager-jdbc:1.0.6'
}

from aws-secretsmanager-jdbc.