Comments (18)
kubernetes/minikube#1001 indicates that awsecr-creds
and registry-creds
are the same, so this bug report should be valid.
One thing that I changed into the RC definition which comes with Minikube (I have the latest downloadable release) is to change the region the awsregion
environment variable in the RC definition:
- name: awsaccount
valueFrom:
secretKeyRef:
key: aws-account
name: awsecr-creds
+ - name: awsregion
+ value: us-west-2
EDIT: changing the awsecr-secret image version from the RC definition does not work: the edit is not accepted by K8S.
- image: upmcenterprises/awsecr-creds:1.1
+ image: upmcenterprises/awsecr-creds:1.2
from registry-creds.
For the record, passing the AWS region as an environment parameter did not work: I had to use the args:
section as @stevesloka mentioned in https://stevesloka.com/2016/12/10/awsecr-creds-minikube-addon/
image: upmcenterprises/awsecr-creds:1.1
+ args:
+ - "--aws-region=eu-west-2"
imagePullPolicy: Always
name: awsecr-creds
With the above, awsecr-creds still SEGV's:
2017-01-23T10:42:01.959722489Z 2017/01/23 10:42:01 Starting up...
2017-01-23T10:42:01.959894422Z 2017/01/23 10:42:01 Using AWS Account: ccp-dev
2017-01-23T10:42:01.959916686Z 2017/01/23 10:42:01 Refresh Interval (minutes): 715
2017-01-23T10:42:02.236912794Z RequestError: send request failed
2017-01-23T10:42:02.236942822Z caused by: Post https://ecr.eu-west-2.amazonaws.com/: net/http: invalid header field value "AWS4-HMAC-SHA256 Credential=AKIAISIZYQJFB6ETYFVQ\n/20170123/eu-west-2/ecr/aws4_request, SignedHeaders=content-length;content-type;host;x-amz-date;x-amz-target, Signature=af5038a8a27f4bd7be9743201e1c8579e7b316cb4929423ddb21a90790c3121c" for key Authorization
2017-01-23T10:42:02.240323707Z panic: runtime error: invalid memory address or nil pointer dereference
2017-01-23T10:42:02.240347982Z [signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x4015b1]
2017-01-23T10:42:02.240353447Z
2017-01-23T10:42:02.240357768Z goroutine 1 [running]:
2017-01-23T10:42:02.240361860Z panic(0x13bb740, 0xc420014020)
2017-01-23T10:42:02.240366011Z /usr/local/go/src/runtime/panic.go:500 +0x1a1
2017-01-23T10:42:02.240370408Z main.process()
2017-01-23T10:42:02.240374434Z /Users/slokas/godev/src/enterprises.io/awsecr-creds/aws_credentials.go:106 +0x91
2017-01-23T10:42:02.240378658Z main.main()
2017-01-23T10:42:02.240386513Z /Users/slokas/godev/src/enterprises.io/awsecr-creds/aws_credentials.go:174 +0x3b1
from registry-creds.
Hey @asnowfix sorry for the confusion. This is the correct repo, it was renamed to registry-creds
since it now handles both AWS and GCR.
It looks like in your logs that you have ccp-dev
for your account id, can you update that with the 8 digit aws accountid and try again?
from registry-creds.
Hello @stevesloka thanks for the reply. I have a 12-digits number in my user's ARN. That same number also shows as my Account Number in the Support Center. Appart from that, I have the account name ccp-dev
. Is this 12-digits number the one you expect? I tried it without more success (message is "server refused for an unknown reason"). I will try to gather more details.
In case the region matters... each time I run kubectl edit rc awsecr-creds -n kube-system
, the added args:
section was removed & I need to re-enter it.
from registry-creds.
Ok so a couple things to try:
- Update to the latest image:
upmcenterprises/registry-creds:1.4
- Make sure the secret referencing
aws-account
is the 8 digit AWS account id (ccp-dev
is not a valid account id from AWS): https://github.com/upmc-enterprises/registry-creds/blob/master/k8s/secret.yaml#L6 - The region does matter if you are trying to get credentials for a region other an
us-east-1
Give that a shot and if not we'll dig some more!
from registry-creds.
Ok, seems that I cannot do much progress easily...
-
It looks like the add-on RC is overwritten every few seconds: any change I do (command-line or dashboard) is reverted quickly.
$ minikube ssh docker pull upmcenterprises/registry-creds:1.4 $ kubectl edit rc awsecr-creds -n kube-system
- name: awsaccount
valueFrom:
secretKeyRef:
key: aws-account
name: awsecr-creds
+ - name: awsregion
+ value: us-west-2
- image: upmcenterprises/awsecr-creds:1.1
+ image: upmcenterprises/registry-creds:1.4
imagePullPolicy: Always
-
Then look at the image value:
$ EDITOR=cat kubectl edit rc awsecr-creds -n kube-system | grep image: image: upmcenterprises/registry-creds:1.4
-
... wait less then 30 secs ...
$ EDITOR=cat kubectl edit rc awsecr-creds -n kube-system | grep image: image: upmcenterprises/awsecr-creds:1.1
- I am not an admin for the
ccp-dev
account (just a developer), I will try to get this value from my admin today - See (1): the region does not last as the RC is reverted by Minikube.
Related to (1) and (3) above & according to https://github.com/kubernetes/minikube/blob/master/ADD_ADDON.md it seems like I need to rebuild Minikube to add a new add-on. Does this apply to a version update?
from registry-creds.
So I found out why you can't edit the replication controller directly, it's being managed by the pod kube-addon-manager-minikube
. To edit do the following:
- minikube ssh
- cd /etc/kubernetes/addons
- sudo vi awsecr-creds-rc.yaml and edit the image to
upmcenterprises/registry-creds:1.4
Can you give that a shot?
from registry-creds.
Thanks for the tip! Will give a try.
Wrt/ the AWS digit numbers, it turns out that we have an account type that one has 12-digits account numbers: according to my account admin, this is related to the fact that he can create sub-accounts for many developers all related to the same billing.
from registry-creds.
Here is the outcome of the test:
Edit: minikube ssh sudo vi /etc/kubernetes/addons/awsecr-creds-rc.yaml
:
- - image: upmcenterprises/awsecr-creds:1.1
+ - image: upmcenterprises/registry-creds:1.4
I noticed that minikube addons disable awsecr-creds
followed by minikube addons enable awsecr-creds
reset that change.
As a result of this change, the extension does not crash:
2017-01-28T10:08:52.043322094Z 2017/01/28 10:08:52 Starting up...
2017-01-28T10:08:52.043371475Z 2017/01/28 10:08:52 Using AWS Account: 957591566260
2017-01-28T10:08:52.043377758Z 2017/01/28 10:08:52 Using AWS Region: us-east-1
2017-01-28T10:08:52.043382207Z 2017/01/28 10:08:52 Refresh Interval (minutes): 60
I will now check if the credential is usable.
EDIT: not the accurate change I did.
from registry-creds.
Thank you for your continuous help
Populating the desired region in the secret & changing the ReplicaSet definition using vi
:
+ - name: awsregion
+ valueFrom:
+ secretKeyRef:
+ name: awsecr-creds
+ key: aws-region
That results in the following (desired) Pod log:
2017-01-28T10:59:41.323068560Z 2017/01/28 10:59:41 Starting up...
2017-01-28T10:59:41.323120511Z 2017/01/28 10:59:41 Using AWS Account: 957591566260
2017-01-28T10:59:41.323134487Z 2017/01/28 10:59:41 Using AWS Region: us-west-2
2017-01-28T10:59:41.323212421Z 2017/01/28 10:59:41 Refresh Interval (minutes): 60
...but when deploying my Pod, the image can still not be pulled from ECR:
$ kubectl get pods ems-874416145-ptfvr
NAME READY STATUS RESTARTS AGE
ems-874416145-ptfvr 1/2 ErrImagePull 0 4m
...and I did not find any more descriptive error message from the registry-creds
add-on: I looked into its Pods log (like above). Is there any other place I should look at?
from registry-creds.
If you use the aws-cli with the same creds, can you pull images on your local machine? Just want to rule out the credentials work.
from registry-creds.
Yes I can. BTW, I do not need to provide my 12-digits account number for that:
$ rm ~/.aws
$ aws configure
AWS Access Key ID [None]: AK...VQ
AWS Secret Access Key [None]: x5...9I
Default region name [None]: us-west-2
Default output format [None]:
Login into AWS:
$ $(aws ecr get-login --region us-west-2)
Flag --email has been deprecated, will be removed in 1.14.
Login Succeeded
Then pulling the image (I took an intermediate snapshot):
$ docker pull 957591566260.dkr.ecr.us-west-2.amazonaws.com/cwp/apigee-edgemicro:latest
latest: Pulling from cwp/apigee-edgemicro
b7f33cc0b48e: Pull complete
e716869be75a: Downloading [====================================> ] 7.414 MB/10.13 MB
a1c4b5a386a0: Downloading [====> ] 4.855 MB/56.83 MB
d34bb69950b9: Download complete
7a9625de0362: Download complete
4ff1bd6a9238: Download complete
6be64de065b3: Download complete
d6f9a928203d: Download complete
c82976e79f0a: Download complete
560caa9c0402: Download complete
from registry-creds.
Hey @asnowfix is this still a problem for you?
from registry-creds.
Yes @stevesloka it is: I had to give-up using minikube due to this in the past few days, but I would love getting it to work again (otherwise I am force to use a shared cluster). If you can publish an images 1.4.1 / whatever with the necessary traces to help troubleshooting this, that would probably be helpful.
from registry-creds.
Hello @stevesloka I resumed testing today with Minikube 0.16.0 & registry-creds 1.5.
- name: awsaccount
valueFrom:
secretKeyRef:
key: aws-account
name: registry-creds
+ - name: awsregion
+ valueFrom:
+ secretKeyRef:
+ key: aws-region
+ name: registry-creds
(BTW it would be nice to have the above change permanent)
...then I kill the live pod so that the RC recreates it:
$ kubectl get pod -n kube-system | grep ^registry-creds | awk '{print $1}' | xargs kubectl delete pod -n kube-system
pod "registry-creds-bcts6" deleted
The logs shows that the region is taken into account, but authentication still fails.
2017-02-24T11:10:57.101181362Z 2017/02/24 11:10:57 Starting up...
2017-02-24T11:10:57.101233474Z 2017/02/24 11:10:57 Using AWS Account: 957591566260
2017-02-24T11:10:57.101241874Z 2017/02/24 11:10:57 Using AWS Region: us-west-2
2017-02-24T11:10:57.101247705Z 2017/02/24 11:10:57 Refresh Interval (minutes): 60
2017-02-24T11:10:57.395997556Z RequestError: send request failed
2017-02-24T11:10:57.396038603Z caused by: Post https://ecr.us-west-2.amazonaws.com/: net/http: invalid header field value "AWS4-HMAC-SHA256 Credential=AKIAISIZYQJFB6ETYFVQ\n/20170224/us-west-2/ecr/aws4_request, SignedHeaders=content-length;content-type;host;x-amz-date;x-amz-target, Signature=4faf3e64bd149cdd7fde030179c19a6dc7537d9d229b3b466a98fbf8b2d74ce7" for key Authorization
from registry-creds.
Hey @asnowfix, sorry this has been so difficult for you to get started. I just added a PR which is now included in the latest minikube version (0.17). Can you download that and give it a try?
When you enable the addon, it now prompts you for the correct pieces needed to start the addon for AWS ro GCR.
Let me know how it works out for you!
from registry-creds.
Problem fixed with minikube 0.17 :-)
BTW, it would be nice to use by default the values from ~/.aws/config
, which are created by a call to aws configure
:
$ cat ~/.aws/config
[default]
aws_access_key_id = A*********************Q
aws_secret_access_key = x*********************************************I
region = us-west-2
from registry-creds.
Nice to hear that it's all resolved and working! I actually had thought about that as well, I've opened an issue to track and integrate support (#41).
from registry-creds.
Related Issues (20)
- Allow running without updating service accounts
- Add ClusterRole, ClusterRoleBinding and ServiceAccount to examples HOT 2
- Allow configuring which secrets it'll try to refresh
- Allow whitelist and blacklist of namespaces
- Empty ecr imagepullsecret
- Intorduce a Helm chart for simplified deployments HOT 4
- Support for aarch64/arm64
- Discrepancy between code and doc
- ERROR creating `registry-creds-dpr` secret minikube addon
- MFA usage with ECR
- Newly published registry-creds image (1.10) does not contain ACR (Azure) support HOT 4
- ECR - InvalidParameterException: Invalid parameter at 'registryIds' failed to satisfy constraint HOT 1
- Q: Enable only one registry HOT 1
- ECR Assume Role issues HOT 1
- client-go: No cached connection was available HOT 14
- Missing File HOT 1
- Allow blacklisting of namespaces HOT 1
- imagePullSecrets not created in all namespaces HOT 1
- v1. Namespace is forbidden: User cannot list resource "namespaces" in API group. HOT 10
- "aws-secret-name" value only set to default HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from registry-creds.