signal SIGSEGV when enabling add-on about registry-creds HOT 18 CLOSED

kubernetes/minikube#1001 indicates that awsecr-creds and registry-creds are the same, so this bug report should be valid.

One thing that I changed into the RC definition which comes with Minikube (I have the latest downloadable release) is to change the region the awsregion environment variable in the RC definition:

        - name: awsaccount
          valueFrom:
            secretKeyRef:
              key: aws-account
              name: awsecr-creds
+       - name: awsregion
+         value: us-west-2

EDIT: changing the awsecr-secret image version from the RC definition does not work: the edit is not accepted by K8S.

-      image: upmcenterprises/awsecr-creds:1.1
+      image: upmcenterprises/awsecr-creds:1.2

from registry-creds.

For the record, passing the AWS region as an environment parameter did not work: I had to use the args: section as @stevesloka mentioned in https://stevesloka.com/2016/12/10/awsecr-creds-minikube-addon/

        image: upmcenterprises/awsecr-creds:1.1
+       args: 
+       - "--aws-region=eu-west-2"
        imagePullPolicy: Always
        name: awsecr-creds

With the above, awsecr-creds still SEGV's:

2017-01-23T10:42:01.959722489Z 2017/01/23 10:42:01 Starting up...
2017-01-23T10:42:01.959894422Z 2017/01/23 10:42:01 Using AWS Account: ccp-dev
2017-01-23T10:42:01.959916686Z 2017/01/23 10:42:01 Refresh Interval (minutes): 715
2017-01-23T10:42:02.236912794Z RequestError: send request failed
2017-01-23T10:42:02.236942822Z caused by: Post https://ecr.eu-west-2.amazonaws.com/: net/http: invalid header field value "AWS4-HMAC-SHA256 Credential=AKIAISIZYQJFB6ETYFVQ\n/20170123/eu-west-2/ecr/aws4_request, SignedHeaders=content-length;content-type;host;x-amz-date;x-amz-target, Signature=af5038a8a27f4bd7be9743201e1c8579e7b316cb4929423ddb21a90790c3121c" for key Authorization
2017-01-23T10:42:02.240323707Z panic: runtime error: invalid memory address or nil pointer dereference
2017-01-23T10:42:02.240347982Z [signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x4015b1]
2017-01-23T10:42:02.240353447Z 
2017-01-23T10:42:02.240357768Z goroutine 1 [running]:
2017-01-23T10:42:02.240361860Z panic(0x13bb740, 0xc420014020)
2017-01-23T10:42:02.240366011Z 	/usr/local/go/src/runtime/panic.go:500 +0x1a1
2017-01-23T10:42:02.240370408Z main.process()
2017-01-23T10:42:02.240374434Z 	/Users/slokas/godev/src/enterprises.io/awsecr-creds/aws_credentials.go:106 +0x91
2017-01-23T10:42:02.240378658Z main.main()
2017-01-23T10:42:02.240386513Z 	/Users/slokas/godev/src/enterprises.io/awsecr-creds/aws_credentials.go:174 +0x3b1

from registry-creds.

Hey @asnowfix sorry for the confusion. This is the correct repo, it was renamed to registry-creds since it now handles both AWS and GCR.

It looks like in your logs that you have ccp-dev for your account id, can you update that with the 8 digit aws accountid and try again?

from registry-creds.

Hello @stevesloka thanks for the reply. I have a 12-digits number in my user's ARN. That same number also shows as my Account Number in the Support Center. Appart from that, I have the account name ccp-dev. Is this 12-digits number the one you expect? I tried it without more success (message is "server refused for an unknown reason"). I will try to gather more details.

In case the region matters... each time I run kubectl edit rc awsecr-creds -n kube-system, the added args: section was removed & I need to re-enter it.

from registry-creds.

Ok so a couple things to try:

1. Update to the latest image: upmcenterprises/registry-creds:1.4
2. Make sure the secret referencing aws-account is the 8 digit AWS account id (ccp-dev is not a valid account id from AWS): https://github.com/upmc-enterprises/registry-creds/blob/master/k8s/secret.yaml#L6
3. The region does matter if you are trying to get credentials for a region other an us-east-1

Give that a shot and if not we'll dig some more!

from registry-creds.

Ok, seems that I cannot do much progress easily...

1. It looks like the add-on RC is overwritten every few seconds: any change I do (command-line or dashboard) is reverted quickly.

    $ minikube ssh docker pull upmcenterprises/registry-creds:1.4
    $ kubectl edit rc awsecr-creds -n kube-system
   

        - name: awsaccount
          valueFrom:
            secretKeyRef:
              key: aws-account
              name: awsecr-creds
+       - name: awsregion
+         value: us-west-2
-       image: upmcenterprises/awsecr-creds:1.1
+       image: upmcenterprises/registry-creds:1.4
        imagePullPolicy: Always

• Then look at the image value:

    $ EDITOR=cat kubectl edit rc awsecr-creds -n kube-system | grep image: 
    image: upmcenterprises/registry-creds:1.4
  

• ... wait less then 30 secs ...

    $ EDITOR=cat kubectl edit rc awsecr-creds -n kube-system | grep image: 
    image: upmcenterprises/awsecr-creds:1.1
  

2. I am not an admin for the ccp-dev account (just a developer), I will try to get this value from my admin today
3. See (1): the region does not last as the RC is reverted by Minikube.

Related to (1) and (3) above & according to https://github.com/kubernetes/minikube/blob/master/ADD_ADDON.md it seems like I need to rebuild Minikube to add a new add-on. Does this apply to a version update?

from registry-creds.

So I found out why you can't edit the replication controller directly, it's being managed by the pod kube-addon-manager-minikube. To edit do the following:

1. minikube ssh
2. cd /etc/kubernetes/addons
3. sudo vi awsecr-creds-rc.yaml and edit the image to upmcenterprises/registry-creds:1.4

Can you give that a shot?

from registry-creds.

Thanks for the tip! Will give a try.

Wrt/ the AWS digit numbers, it turns out that we have an account type that one has 12-digits account numbers: according to my account admin, this is related to the fact that he can create sub-accounts for many developers all related to the same billing.

from registry-creds.

Here is the outcome of the test:

Edit: minikube ssh sudo vi /etc/kubernetes/addons/awsecr-creds-rc.yaml:

-     - image: upmcenterprises/awsecr-creds:1.1
+    - image: upmcenterprises/registry-creds:1.4

I noticed that minikube addons disable awsecr-creds followed by minikube addons enable awsecr-creds reset that change.

As a result of this change, the extension does not crash:

2017-01-28T10:08:52.043322094Z 2017/01/28 10:08:52 Starting up...
2017-01-28T10:08:52.043371475Z 2017/01/28 10:08:52 Using AWS Account: 957591566260
2017-01-28T10:08:52.043377758Z 2017/01/28 10:08:52 Using AWS Region: us-east-1
2017-01-28T10:08:52.043382207Z 2017/01/28 10:08:52 Refresh Interval (minutes): 60

I will now check if the credential is usable.

EDIT: not the accurate change I did.

from registry-creds.

Thank you for your continuous help

Populating the desired region in the secret & changing the ReplicaSet definition using vi:

+          - name: awsregion               
+            valueFrom:                    
+              secretKeyRef:               
+                name: awsecr-creds
+                key: aws-region   

That results in the following (desired) Pod log:

2017-01-28T10:59:41.323068560Z 2017/01/28 10:59:41 Starting up...
2017-01-28T10:59:41.323120511Z 2017/01/28 10:59:41 Using AWS Account: 957591566260
2017-01-28T10:59:41.323134487Z 2017/01/28 10:59:41 Using AWS Region: us-west-2
2017-01-28T10:59:41.323212421Z 2017/01/28 10:59:41 Refresh Interval (minutes): 60

...but when deploying my Pod, the image can still not be pulled from ECR:

$ kubectl get pods ems-874416145-ptfvr
NAME                  READY     STATUS         RESTARTS   AGE
ems-874416145-ptfvr   1/2       ErrImagePull   0          4m

...and I did not find any more descriptive error message from the registry-creds add-on: I looked into its Pods log (like above). Is there any other place I should look at?

from registry-creds.

If you use the aws-cli with the same creds, can you pull images on your local machine? Just want to rule out the credentials work.

from registry-creds.

Yes I can. BTW, I do not need to provide my 12-digits account number for that:

$ rm ~/.aws
$ aws configure
AWS Access Key ID [None]: AK...VQ
AWS Secret Access Key [None]: x5...9I
Default region name [None]: us-west-2
Default output format [None]:

Login into AWS:

$ $(aws ecr get-login --region us-west-2)
Flag --email has been deprecated, will be removed in 1.14.
Login Succeeded

Then pulling the image (I took an intermediate snapshot):

$ docker pull 957591566260.dkr.ecr.us-west-2.amazonaws.com/cwp/apigee-edgemicro:latest
latest: Pulling from cwp/apigee-edgemicro
b7f33cc0b48e: Pull complete 
e716869be75a: Downloading [====================================>              ] 7.414 MB/10.13 MB
a1c4b5a386a0: Downloading [====>                                              ] 4.855 MB/56.83 MB
d34bb69950b9: Download complete 
7a9625de0362: Download complete 
4ff1bd6a9238: Download complete 
6be64de065b3: Download complete 
d6f9a928203d: Download complete 
c82976e79f0a: Download complete 
560caa9c0402: Download complete 

from registry-creds.

Hey @asnowfix is this still a problem for you?

from registry-creds.

Yes @stevesloka it is: I had to give-up using minikube due to this in the past few days, but I would love getting it to work again (otherwise I am force to use a shared cluster). If you can publish an images 1.4.1 / whatever with the necessary traces to help troubleshooting this, that would probably be helpful.

from registry-creds.

Hello @stevesloka I resumed testing today with Minikube 0.16.0 & registry-creds 1.5.

          - name: awsaccount
            valueFrom:
              secretKeyRef:
                key: aws-account
                name: registry-creds
+         - name: awsregion
+           valueFrom:
+             secretKeyRef:
+               key: aws-region
+               name: registry-creds

(BTW it would be nice to have the above change permanent)

...then I kill the live pod so that the RC recreates it:

$ kubectl get pod -n kube-system | grep ^registry-creds | awk '{print $1}' | xargs kubectl delete pod -n kube-system
pod "registry-creds-bcts6" deleted

The logs shows that the region is taken into account, but authentication still fails.

2017-02-24T11:10:57.101181362Z 2017/02/24 11:10:57 Starting up...
2017-02-24T11:10:57.101233474Z 2017/02/24 11:10:57 Using AWS Account: 957591566260
2017-02-24T11:10:57.101241874Z 2017/02/24 11:10:57 Using AWS Region: us-west-2
2017-02-24T11:10:57.101247705Z 2017/02/24 11:10:57 Refresh Interval (minutes): 60
2017-02-24T11:10:57.395997556Z RequestError: send request failed
2017-02-24T11:10:57.396038603Z caused by: Post https://ecr.us-west-2.amazonaws.com/: net/http: invalid header field value "AWS4-HMAC-SHA256 Credential=AKIAISIZYQJFB6ETYFVQ\n/20170224/us-west-2/ecr/aws4_request, SignedHeaders=content-length;content-type;host;x-amz-date;x-amz-target, Signature=4faf3e64bd149cdd7fde030179c19a6dc7537d9d229b3b466a98fbf8b2d74ce7" for key Authorization

from registry-creds.

Hey @asnowfix, sorry this has been so difficult for you to get started. I just added a PR which is now included in the latest minikube version (0.17). Can you download that and give it a try?

When you enable the addon, it now prompts you for the correct pieces needed to start the addon for AWS ro GCR.

Let me know how it works out for you!

from registry-creds.

Problem fixed with minikube 0.17 :-)

BTW, it would be nice to use by default the values from ~/.aws/config, which are created by a call to aws configure:

$ cat ~/.aws/config 
[default]
aws_access_key_id = A*********************Q
aws_secret_access_key = x*********************************************I
region = us-west-2

from registry-creds.

Nice to hear that it's all resolved and working! I actually had thought about that as well, I've opened an issue to track and integrate support (#41).

from registry-creds.