Comments (7)
amitkarpe
commented on June 19, 2024
1
Hi,
Looks like init container did not run and so secrets-init-container is missing in the pod description.
➜ kubectl describe pod -l run=webserver
Name: webserver-888fc6786-4z7hp
Namespace: secret
Priority: 0
Node: ip-10-23-7-28.ap-southeast-1.compute.internal/10.23.7.28
Start Time: Wed, 12 Aug 2020 14:14:50 +0800
Labels: pod-template-hash=888fc6786
run=webserver
Annotations: kubernetes.io/psp: eks.privileged
secrets.k8s.aws/secret-arn: arn:aws:secretsmanager:ap-southeast-1:123456789012:secret:test_secretB-wFblqy
secrets.k8s.aws/sidecarInjectorWebhook: enabled
Status: Running
IP: 10.23.6.103
IPs: <none>
Controlled By: ReplicaSet/webserver-888fc6786
Containers:
webserver:
Container ID: docker://b47228f01099a63006aba623a2c99966432baca592b51c73f5979124337117b5
Image: busybox:1.28
Image ID: docker-pullable://busybox@sha256:141c253bc4c3fd0a201d32dc1f493bcf3fff003b6df416dea4f41046e0f37d47
Port: <none>
Host Port: <none>
Command:
sh
-c
echo $(cat /tmp/secret) && sleep 3600
State: Running
Started: Wed, 12 Aug 2020 14:14:51 +0800
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-d5cwh (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
default-token-d5cwh:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-d5cwh
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 15m default-scheduler Successfully assigned secret/webserver-888fc6786-4z7hp to ip-10-23-7-28.ap-southeast-1.compute.internal
Normal Pulled 15m kubelet, ip-10-23-7-28.ap-southeast-1.compute.internal Container image "busybox:1.28" already present on machine
Normal Created 15m kubelet, ip-10-23-7-28.ap-southeast-1.compute.internal Created container webserver
Normal Started 15m kubelet, ip-10-23-7-28.ap-southeast-1.compute.internal Started container webserver
from aws-secret-sidecar-injector.
amitkarpe
commented on June 19, 2024
1
Hi @antonosmond ,
I tried with default namespace still, I am facing the same issue.
from aws-secret-sidecar-injector.
antonosmond
commented on June 19, 2024
@amitkarpe You have the same problem I did. See #24.
Until #21 is merged, secret-inject must be deployed in the default namespace.
from aws-secret-sidecar-injector.
amitkarpe
commented on June 19, 2024
Hi @jicowan,
I have created new EKS cluster. I used default namespace.
I was able to see secrets-init-container as "Init Containers". But can got access /tmp/secret.
➜ kl -l app=webserver -f
cat: can't open '/tmp/secret': No such file or directory
^C
Deployment Description
➜ kdp -l app=webserver [52/577]
Name: webserver-7c597ffbfc-qx7kf
Namespace: default
Priority: 0
Node: ip-192-168-252-96.ec2.internal/192.168.252.96
Start Time: Fri, 14 Aug 2020 12:48:10 +0800
Labels: app=webserver
pod-template-hash=7c597ffbfc
run=webserver
Annotations: kubernetes.io/psp: eks.privileged
secrets.k8s.aws/secret-arn: arn:aws:secretsmanager:us-east-1:333438771545:secret:mysec-FfLLqF
secrets.k8s.aws/sidecarInjectorWebhook: enabled
Status: Running
IP: 192.168.193.231
IPs:
IP: 192.168.193.231
Controlled By: ReplicaSet/webserver-7c597ffbfc
Init Containers:
secrets-init-container:
Container ID: docker://3fe84c1ca8957b833723907f13b061ad4be02dc651290f444353a796dfd65060
Image: docker.io/amazon/aws-secrets-manager-secret-sidecar:v0.1.1
Image ID: docker-pullable://amazon/aws-secrets-manager-secret-sidecar@sha256:079028dde7738a622ecbf35ee6b345bba7a0a5f40584c4e97e21535ced0cc421
Port: <none>
Host Port: <none>
State: Terminated
Reason: Completed
Exit Code: 0
Started: Fri, 14 Aug 2020 12:48:18 +0800
Finished: Fri, 14 Aug 2020 12:48:18 +0800
Ready: True
Restart Count: 0
Environment:
SECRET_ARN: (v1:metadata.annotations['secrets.k8s.aws/secret-arn'])
AWS_ROLE_ARN: arn:aws:iam::333438771545:role/test_sec
AWS_WEB_IDENTITY_TOKEN_FILE: /var/run/secrets/eks.amazonaws.com/serviceaccount/token
Mounts:
/tmp from secret-vol (rw)
/var/run/secrets/eks.amazonaws.com/serviceaccount from aws-iam-token (ro)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-k8jgl (ro)
Containers:
webserver:
Container ID: docker://60ed9642c55aba1a593c286e6a73ddc6281f6d0e9d6e7551a2c1d18c2521df8b
Image: amitkarpe/aws-cli
Image ID: docker-pullable://amitkarpe/aws-cli@sha256:2cd871e804eb7d3ae645129079174c51de102d4d91fc6781c22df74c439f320d
Port: <none>
Host Port: <none>
Command:
sh
-c
echo $(cat /tmp/secret) && sleep 3600
State: Running
Started: Fri, 14 Aug 2020 12:48:24 +0800
Ready: True
Restart Count: 0
Environment:
AWS_ROLE_ARN: arn:aws:iam::333438771545:role/test_sec
AWS_WEB_IDENTITY_TOKEN_FILE: /var/run/secrets/eks.amazonaws.com/serviceaccount/token
Mounts:
/tmp/ from secret-vol (rw)
/var/run/secrets/eks.amazonaws.com/serviceaccount from aws-iam-token (ro)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-k8jgl (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
aws-iam-token:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 86400
default-token-k8jgl:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-k8jgl
Optional: false
secret-vol:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium: Memory
SizeLimit: <unset>
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 27s default-scheduler Successfully assigned default/webserver-7c597ffbfc-qx7kf to ip-192-168-252-96.ec2.internal
Normal Pulling 26s kubelet, ip-192-168-252-96.ec2.internal Pulling image "docker.io/amazon/aws-secrets-manager-secret-sidecar:v0.1.1"
Normal Pulled 20s kubelet, ip-192-168-252-96.ec2.internal Successfully pulled image "docker.io/amazon/aws-secrets-manager-secret-sidecar:v0.1.1"
Normal Created 19s kubelet, ip-192-168-252-96.ec2.internal Created container secrets-init-container
Normal Started 19s kubelet, ip-192-168-252-96.ec2.internal Started container secrets-init-container
Normal Pulling 18s kubelet, ip-192-168-252-96.ec2.internal Pulling image "amitkarpe/aws-cli"
Normal Pulled 14s kubelet, ip-192-168-252-96.ec2.internal Successfully pulled image "amitkarpe/aws-cli"
Normal Created 14s kubelet, ip-192-168-252-96.ec2.internal Created container webserver
Normal Started 13s kubelet, ip-192-168-252-96.ec2.internal Started container webserver
Pod status
status: [4/970]
conditions:
- lastProbeTime: null
lastTransitionTime: "2020-08-14T04:48:19Z"
status: "True"
type: Initialized
- lastProbeTime: null
lastTransitionTime: "2020-08-14T04:48:24Z"
status: "True"
type: Ready
- lastProbeTime: null
lastTransitionTime: "2020-08-14T04:48:24Z"
status: "True"
type: ContainersReady
- lastProbeTime: null
lastTransitionTime: "2020-08-14T04:48:10Z"
status: "True"
type: PodScheduled
containerStatuses:
- containerID: docker://60ed9642c55aba1a593c286e6a73ddc6281f6d0e9d6e7551a2c1d18c2521df8b
image: amitkarpe/aws-cli:latest
imageID: docker-pullable://amitkarpe/aws-cli@sha256:2cd871e804eb7d3ae645129079174c51de102d4d91fc6781c22df74c439f320d
lastState: {}
name: webserver
ready: true
restartCount: 0
started: true
state:
running:
startedAt: "2020-08-14T04:48:24Z"
hostIP: 192.168.252.96
initContainerStatuses:
- containerID: docker://3fe84c1ca8957b833723907f13b061ad4be02dc651290f444353a796dfd65060
image: amazon/aws-secrets-manager-secret-sidecar:v0.1.1
imageID: docker-pullable://amazon/aws-secrets-manager-secret-sidecar@sha256:079028dde7738a622ecbf35ee6b345bba7a0a5f40584c4e97e21535ced0cc421
lastState: {}
name: secrets-init-container
ready: true
restartCount: 0
state:
terminated:
containerID: docker://3fe84c1ca8957b833723907f13b061ad4be02dc651290f444353a796dfd65060
exitCode: 0
finishedAt: "2020-08-14T04:48:18Z"
reason: Completed
startedAt: "2020-08-14T04:48:18Z"
phase: Running
podIP: 192.168.193.231
podIPs:
- ip: 192.168.193.231
qosClass: BestEffort
startTime: "2020-08-14T04:48:10Z"
from aws-secret-sidecar-injector.
amitkarpe
commented on June 19, 2024
Hi,
When I tried to access s3 or SM, I got following error:
➜ export POD=$(kubectl get pods -l "app=${app}" -o jsonpath="{.items[0].metadata.name}"); kubectl exec -it $POD -- sh
/ # aws secretsmanager get-secret-value --secret-id mysec --region us-east-1
An error occurred (AccessDenied) when calling the AssumeRoleWithWebIdentity operation: Not authorized to perform sts:AssumeRoleWithWebIdentity
/ # aws s3 ls
An error occurred (AccessDenied) when calling the AssumeRoleWithWebIdentity operation: Not authorized to perform sts:AssumeRoleWithWebIdentity
/ #
from aws-secret-sidecar-injector.
jicowan
commented on June 19, 2024
Looks like the service account you are referencing in your pod.spec doesn't have permission to read the secret from Secrets Manager. If the SA is not mapped to a role that has access to your secret, the init container will not be able to write the secret to the volume.
from aws-secret-sidecar-injector.
amitkarpe
commented on June 19, 2024
@jicowan
Thank you for your feedback.
I was not using the correct Trust Relationship. After using more generic like "system:serviceaccount:*" and "StringLike" instead of "StringEquals", I was able to solve the problem.
from aws-secret-sidecar-injector.
Related Issues (20)
- Feature request - inject secrets as env vars HOT 3
- WebIdentity Error HOT 1
- [Question] Deploying secret-inject Controller in another namespace HOT 4
- Production readiness HOT 1
- Inject as an Environment variable HOT 1
- Changing mount point and ecrousseau/aws-secret-injector fork. HOT 3
- change secret mount point inside the container HOT 6
- Feature Request: inject secrets into init containers HOT 2
- Feature request - support multiple Secrets injection HOT 15
- Feature Request - AWS Parameter Store Integration HOT 3
- Init container isn't started for webserver + TLS handshake error in secret-inject HOT 11
- How to deserialize the json data that comes from SecretsManager and got loaded in the volumeMount HOT 9
- http: TLS handshake error from <IP>:<PORT>: remote error: tls: bad certificate HOT 4
- initContainers are overwritten in deployment HOT 2
- add operation does not apply: doc is missing path: "/spec/initContainers/0 HOT 1
- ability to create serviceAccount by using helm HOT 1
- Latest chart (0.1.7) errors on install HOT 1
- Inject secret as environment variable HOT 1
- set_aws_parameters does not set OPERATOR_REGION, SQS_URL, IAM_ARN
- system-manager inject HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from aws-secret-sidecar-injector.