EKS

Implementation of EKS setup using Terraform and Cloudformation. Fully functional templates to deploy your VPC and Kubernetes clusters together with all the essential tags and addons. Also, worker nodes are part of AutoScallingGroup which consists of spot and on-demand instances.

Templates support deployment to different AWS partitions. I have tested it with public and china partitions. I am actively using this configuration to run EKS setup in Ireland(eu-west-1), North Virginia(us-east-1) and Beijing(cn-north-1).

terraform-aws and terraform-k8s templates

Latest configuration templates used by me can be found in terraform-aws for aws provider and terraform-k8s for kubernetes provider. Once you configure your environment variables in ./terraform-aws/vars ./terraform-k8s/vars, you can use makefile commands to run your deployments. Resources that will be created after applying templates:

You will find latest setup of following components:

1. VPC with public/private subnets, enabled flow logs and VPC endpoints for ECR and S3
2. EKS controlplane
3. EKS worker nodes in private subnets (spot and ondemnd instances based on variables)
4. Option to used Managed Node Groups
5. Dynamic basion host
6. Automatically configure aws-auth configmap for worker nodes to join the cluster
7. OpenID Connect provider which can be used to assign IAM roles to service accounts in k8s
8. NodeDrainer lambda which will drain worker nodes during rollingUpdate of the nodes (This is only applicable to spot worker nodes, managed node groups do not require this lambda). Node drainer lambda is maintained in https://github.com/marcincuber/tf-k8s-node-drainer
9. IAM Roles for service accounts such as aws-node, cluster-autoscaler, alb-ingress-controller, external-secrets (Role arns are used when you deploy kubernetes addons with Service Accounts that make use of OIDC provider)
10. For spot termination handling use aws-node-termination-handler from k8s_templates/aws-node-termination-handler.
11. EKS cluster add-ons (CoreDNS + kube-proxy)

Kubernetes YAML templates

All the templates for additional deployments/daemonsets can be found in k8s_templates.

To apply templates simply run kubectl apply -f . from a desired folder. Ensure to put in correct Role arn in service accounts configuration. Also, check that environment variables are correct.

You will find templates for the following Kubernetes components:

• ALB ingress controller
• AWS Load Balancer controller
• AWS node termination handler
• Calico
• Cert Manager
• Cluster Autoscaler
• CoreDns
• Dashboard
• External-DNS
• External Secrets
• Kube Proxy
• Kube2iam
• Metrics server
• NewRelic
• Reloader
• Spot Interrupt Handler
• VPC CNI Plugin
• Secrets CSI Driver

Docs and other additional resources

Check out my stories on medium if you interested in finding out more on specific topics.

Amazon EKS upgrade 1.20 to 1.21

Amazon EKS upgrade journey from 1.20 to 1.21

Amazon EKS upgrade 1.19 to 1.20

Amazon EKS upgrade journey from 1.19 to 1.20

Amazon EKS upgrade 1.18 to 1.19

Amazon EKS upgrade journey from 1.18 to 1.19

Amazon EKS upgrade 1.17 to 1.18

Amazon EKS upgrade journey from 1.17 to 1.18

Amazon EKS upgrade 1.16 to 1.17

Amazon EKS upgrade journey from 1.16 to 1.17

Amazon EKS upgrade 1.15 to 1.16

Amazon EKS upgrade journey from 1.15 to 1.16

Amazon EKS Addons

Amazon EKS Addons

EKS + Kube-bench

Kube-bench implementation with EKS

Amazon EKS design, use of spot instances and cluster scaling

More about my configuration can be found in the blog post I have written recently -> EKS design

IAM Roles for specific namespaces

Amazon EKS- RBAC with IAM access

IAM Roles for service accounts using OpenID Connect

Using OIDC provider to allow service accounts to assume IAM role

Kube2iam

More about kube2iam configuration can be found in the blog post I have written recently -> EKS and kube2iam

External DNS

Amazon EKS, setup external DNS with OIDC provider and kube2iam

EKS Managed Node Groups

Amazon EKS + managed node groups

Terraform module written by me can be found in -> https://registry.terraform.io/modules/umotif-public/eks-node-group

Gitlab runners on EKS

Kubernetes GitLab Runners on Amazon EKS

Useful resources

EKS platforms information Worker nodes upgrades

Generate kubeconfig file

On user's machine who has been added to EKS, they can configure .kube/config file using the following command:

$ aws eks list-clusters
$ aws eks update-kubeconfig --name ${cluster_name}