Better support for multiple clusters about terraform-aws-eks HOT 7 CLOSED

Hi @max-rocket-internet 🖖

I like these ideas but I'll need some clarification. I'm far more experienced on the terraform bits so excuse any of my ignorance on the k8s side...

I think what you're requesting here is the ability to output multiple configuration files for both kubeconfig and the config-map-aws-auth.yaml. This use case (though somewhat problematic... I'll explain) is the reason I included the config_output_path variable, so that if you really wanted to instantiate the module twice within the same stateful repo, you could output the config files to different sub-directories in the same leaf dir. Does that seem like a reasonable way forward for you?

Maybe you were referring to worker nodes being referenced in the config files? That doesn't quite make sense to me as they're cluster-wide, not worker ASG specific. In either case, the EKS cluster name is referenced in kubeconfig here https://github.com/terraform-aws-modules/terraform-aws-eks/blob/master/templates/kubeconfig.tpl#L27 . I'm not familiar enough with the configs here to know where the cluster name would fit in the config map https://github.com/terraform-aws-modules/terraform-aws-eks/blob/master/templates/config-map-aws-auth.yaml.tpl .

Also, generally when you're instantiating templates in a stateful repo (live/infrastructure repo), it's best to keep state small to limit blast radius, improve apply speed of a state, work with limited dependencies at any given time, etc. To defy my own reasoning for a second, one could argue the opposite on speed in some cases (if I'm instantiating EKS twice and in separate states, parallelism must be handled by the user or the operation is 2x longer). Keeping small state isn't a hard-and-fast best practice but you'll find this is a common pattern in larger deployments with many collaborators.

In short, I'm struggling to imagine a good use case for this EKS module being instantiated n times in the same state location. Perhaps you're using workspaces and getting bit by overriding config but in that case, again pass something unique from the workspace as the config_output_path variable.

If the bits above don't actually speak to your problem, my apologies, I've fumbled! If I've missed the mark it'd be great if you could elaborate on your use case a little more. Thnks for the request and let me know if you're comfortable submitting a PR if there's work to be done. 👍

from terraform-aws-eks.

Hey @brandoconnor,

I'm far more experienced on the terraform bits so excuse any of my ignorance on the k8s side

All good 👍

I think what you're requesting here is the ability to output multiple configuration files for both kubeconfig and the config-map-aws-auth.yaml

Yes this. And making changes to some keys in kubeconfig to make the config unique (will explain more later)

you could output the config files to different sub-directories in the same leaf dir

OK sure but still by default a second cluster will overwrite the file, which is annoying. I think you should by default include some unique value either in the file name or its path. EKS won't support in place upgrades for some times so running 2 clusters side by side will be common, at least in the beginning. Also, some people (like myself), will definitely just want multiple clusters in a single "stateful repo" anyway.

the EKS cluster name is referenced in kubeconfig here https://github.com/terraform-aws-modules/terraform-aws-eks/blob/master/templates/kubeconfig.tpl#L27

It is referenced at that line but that part of the config is not relevant to what I'm talking about. I'll explain....

kubectl by default has a global config file ~/.kube/config. I would say that for 90% of k8s users, all cluster config goes in here in a single file. Many users have multiple clusters in here, like staging, dev, test etc. Here's my example with 4 clusters:

$ kubectl config get-contexts
CURRENT   NAME                                                CLUSTER                                             AUTHINFO                                            NAMESPACE
          cluster1.domain.com                            cluster1.domain.com                            cluster1.domain.com
*         eks_cluster_1                                       eks_cluster_1                                       aws
          gke_balkan-live_europe-west1-d_live-cluster         gke_balkan-live_europe-west1-d_live-cluster         gke_balkan-live_europe-west1-d_live-cluster
          gke_balkan-staging_europe-west1-d_staging-cluster   gke_balkan-staging_europe-west1-d_staging-cluster   gke_balkan-staging_europe-west1-d_staging-cluster

Now look at example output from when I use the current TF generated kubeconfig file:

kubectl config get-contexts
CURRENT   NAME      CLUSTER      AUTHINFO   NAMESPACE
*         aws       kubernetes   aws

See aws and kubernetes strings are not unique. So I have to manually change those fields to the cluster name and then import that config into my ~/.kube/config file. Either way, those strings don't help me to know what cluster I'm connected to because they don't mean anything.

I'm not familiar enough with the configs here to know where the cluster name would fit in the config map https://github.com/terraform-aws-modules/terraform-aws-eks/blob/master/templates/config-map-aws-auth.yaml.tpl .

The contents of the file don't need to change, just the path or filename should be unique by default.

In short, I'm struggling to imagine a good use case for this EKS module being instantiated n times in the same state location

Trust me, this will happen 🙂

from terraform-aws-eks.

Thnks for the request and let me know if you're comfortable submitting a PR if there's work to be done

Yes sure, no problem.

Also worth noting that the bottom 2 clusters in my kubectl config get-contexts output are from GKE, the Google equivalent of EKS. So you can see there how they name the clusters by default.

from terraform-aws-eks.

Got it. That all makes a bit better sense. I appreciate the explanation.

Your proposed change is just prefixing the output config files with ${var.cluster_name}. That makes perfectly good sense. PR away!

from terraform-aws-eks.

I ran into the same issue and made the kubeconfig significantly more configurable in PR #49.

from terraform-aws-eks.

Awesome, thanks @sdavids13 !

Your proposed change is just prefixing the output config files with ${var.cluster_name}. That makes perfectly good sense. PR away!

Perfect.

from terraform-aws-eks.

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

from terraform-aws-eks.