cannot use awsIAM roles instead of aws accessKey about builder HOT 8 OPEN

From @vdice on February 16, 2017 18:12

The yaml file mentions, if you leave it blank it will use IAM roles. I'm not sure it's using the IAM roles because the registry-logs seems to open the dir for creds

I don't have direct experience with failures relating to IAM roles, but my first guess is the issue may lie in IAM role(s)/permissions set up/used by the cluster... Perhaps double-check that using the same role(s)/perms works properly elsewhere (say via aws invocations in a terminal).

from builder.

From @blakebarnett on February 17, 2017 0:15

I ran into this also, I realized it's failing during the build because builder generates a pod spec that doesn't have the IAM role annotation. The dockerbuilder/slugbuilder podspec would need to inherit the AWS role annotation use for kube2iam.

Maybe something like this would work, though it would need to be conditional, etc:

index 8418cc6..d68771f 100644
--- a/pkg/gitreceive/k8s_util.go
+++ b/pkg/gitreceive/k8s_util.go
@@ -28,6 +28,7 @@ const (
        builderStorage   = "BUILDER_STORAGE"
        objectStorePath  = "/var/run/secrets/deis/objectstore/creds"
        envRoot          = "/tmp/env"
+       iamRole          = "IAM_ROLE"
 )

 func dockerBuilderPodName(appName, shortSha string) string {
@@ -166,6 +167,9 @@ func buildPod(
                        Labels: map[string]string{
                                "heritage": name,
                        },
+                       Annotations: map[string]string{
+                               "iam.amazonaws.com/role": iamRole,
+                       },
                },
        }

from builder.

@Akshaykapoor @blakebarnett @vdice Any news on this? We are experiencing the same problem 403 Access Denied but the IAM policies are correct. Seems like deis-builder is not respecting the blank access and secret keys and trying to use them.

from builder.

From @blakebarnett on May 8, 2017 20:28

Sorry, we had to stop using deis workflow because of this and a few other reasons.

from builder.

@blakebarnett, may I ask what the other reasons were?

Just interested in learning what else you ran into that was a deal breaker for you and your team.

from builder.

From @blakebarnett on May 9, 2017 17:26

Here's the non-sugar-coated list.

• Separate authentication scheme
• No support for RBAC or team-based access
• No support for kube2iam for AWS IAM integration
• Single app per-namespace limitation (hard-coded)
• No annotations
• Restricted label support
• No integration with Helm
• Deis router is buggy and slow
• Too opinionated on metrics and logging
• Very slow pace of recent development
• Uncertain future after Microsoft acquisition

from builder.

Thank you for the list @blakebarnett . We were aware with most of these. Agreed that it would be nice to have kube2iam integration for the pods and RBAC is being added in the next version it seems like.

The single app per-namespace is annoying as well.

What did you guys end up using instead or did you just go with some kind of custom kubernetes setup?

from builder.

From @blakebarnett on May 9, 2017 17:58

We're just building everything using CI and Helm charts for now, in hope that at some later point everything will play nicely and we can provide PaaS features.

from builder.