Status:
Currently, AWS EKS is running the worker nodes with the default volume size and encryption isn't enabled.

Goal:
Implement the possibility to support encrypted root device and setting the volume size.

We came into an issue where there was a requirement to use TF 0.13, I would have thought it would have been an easy switch (just switch out the KubeStack Docker image with a Custom one that has TF 0.13 installed). The issue arises that the KubeStack makes use of Kubestacks custom Kustomize Terraform provider which is installed in the same container. The pathway for custom terraform providers has been changed. So you need to dig into Kubestack and pull out the Dockerfile and update it accordingly in order to get it to work.

There where two problems with this:

• The KubeStack Dockerfile setup is quite layered so it was difficult to pinpoint the exact places to make the change
• You lose the ability to just pull Kubestacks image and have to maintain your own

Enhancement Request:

As a user, I would like to configure my Kubestack environment in Azure with CNI/Advanced Networking Config to allow greater control over networking resources.

Background

We are looking to use Kubestack to manage our AKS clusters within a fairly complex hybrid cloud/on-prem environment. Because the on premises resources we need to access from within the cluster are in a 10.10.10.0/24 subnet, and the vnets created automatically by the AKS managed service have an addressable space of 10.0.0.0/8, which contains 10.10.10.0/24, the collision of these address spaces have prevented us from creating a VPN gateway to link the AKS vnet and the local subnet. Using CNI will allow us to create a vnet with a smaller address space that we can configure and manage as needed.

Detail

To enable this behavior, the following changes would be needed:

• The user would create a custom vnet and subnet (outside the Kubestack directory, in a wider TF project?)
• The user would pass the subnet ID in to Kubestack (via config.auto.tfvars?)
• The user would pass in, minimally, values for service_cidr and dns_server_ip (optionally also network_plugin, network_policy, and pod_cidr)

Challenge

In order to complete the CNI setup, the user will need to associate the network security group which is generated by the AKS managed service with the custom subnet. Unfortunately the group name includes a randomly generated number, i.e. aks-agentpool-22342344-nsg, and the azurerm_kubernetes_cluster provider does not expose this group name directly within the Terraform state. It is possible to retrieve it using an external data provider and a brief shell script. To facilitate this, it would be helpful if the Kubestack cluster module could expose the name of the auto-created node_resource_group that contains the NSG.

The kind_cluster name should be lowercase, otherwise apply will be failing:

.terraform/modules/eks_zero/kind/_modules/kind/main.tf line 12, in resource "kind_cluster" "current":
  12: resource "kind_cluster" "current" {

Istio was not adding sidecars to pods. After some digging, I found an issue which mentioned that EKS needs an ingress rule on the workers for port 443 to the masters. Adding this ingress rule to the worker security group allowed the sidecars to be created.

Hello Team ,

https://github.com/kbst/terraform-kubestack/blob/master/aws/_modules/eks/workers_asg.tf

The filter provided for retrieving the AMI for creating EKS Worked node is not working for eu-central-1 region. It's not retrieving any AMI .

aws ec2 --region eu-central-1 describe-images --filters "Name=name,Values=amazon-eks-node*" --owner 602401143452

{
"Images": []
}

When tried the same from us-west-2 regions . It is able to retrieve results.

The reason being . The EKS-worker-node AMIs in eu-central-1 are having the EKS-K8s versions in it .

"Name": "amazon-eks-node-1.11-v20190109"

So the filter value has to be

["amazon-eks-node-${aws_eks_cluster.current.version}-v*"]

{
            "Architecture": "x86_64",
            "CreationDate": "2019-01-10T00:31:41.000Z",
            "ImageId": "ami-010caa98bae9a09e2",
            "ImageLocation": "602401143452/amazon-eks-node-1.11-v20190109",
            "ImageType": "machine",
            "Public": true,
            "OwnerId": "602401143452",
            "State": "available",
            "BlockDeviceMappings": [
                {
                    "DeviceName": "/dev/xvda",
                    "Ebs": {
                        "DeleteOnTermination": true,
                        "SnapshotId": "snap-0cb69b54efae33948",
                        "VolumeSize": 20,
                        "VolumeType": "gp2",
                        "Encrypted": false
                    }
                }
            ],
            "Description": "EKS Kubernetes Worker AMI with AmazonLinux2 image",
            "EnaSupport": true,
            "Hypervisor": "xen",
            "Name": "amazon-eks-node-1.11-v20190109",
            "RootDeviceName": "/dev/xvda",
            "RootDeviceType": "ebs",
            "SriovNetSupport": "simple",
            "VirtualizationType": "hvm"
        }

Please make the corresponding changes so that it works for eu-central-1 region as well.

Using the latest version of the kubernetes provider fails to launch the aws quickstart stack (quickstart/src/configurations/eks) with error:

Error: Failed to initialize config: invalid configuration: no configuration has been provided

This seems to be a known issue - see hashicorp/terraform-provider-kubernetes#759

Pinning the provider at 1.10 seems to be a workaround for the issue:

quickstart/src/configurations/eks/versions.tf
terraform {
  required_version = ">= 0.12"
  required_providers {
    kubernetes = "1.10"
  }
}

Goal of the Kubestack modules is to harmonize the managed K8s solutions across the different cloud providers as much as possible. This should be reflected in the input variables too. Currently EKS has min, max node settings. The GKE module does not currently enable the autoscaler and does not expose the min, max variables.

Currently none of the kubestack modules have any outputs. This might make extending the clusters with custom terraform code difficult if one also wishes not to touch the internals in order to provision custom resources around the cluster.

There's a couple questions really:

1. How do we envision extending the cluster with custom terraform declarations?
2. How do we upgrade the kubestack version?
3. Should we start implementing outputs similar to the output proposed in #133 ?

I envision upgrades being a rather manual process, maybe git merge upstream, keeping kbst/terraform as the upstream.
At the same time I'd envision a single extensions.tf using the proposed outputs in order to extend the clusters with custom terraform declarations.

When the AKS module creates vnet and subnet names, it does not use the metadata name. The generated names only worked for multiple environments in the same Azure resource group. But not for multiple clusters per environment in the same resource group.

Hi there,

Following the quickstart documentation in the step 3. Bootstrap the Ops-and Apps-cluster pair I got the following error:

$ terraform apply --auto-approve

module.gke_zero.module.cluster.data.external.gcloud_account: Refreshing state...
module.gke_zero.module.cluster.data.google_client_config.default: Refreshing state...
module.gke_zero.module.cluster.module.cluster_services.data.kustomization.current: Refreshing state...

Error: Failed to initialize config: invalid configuration: no configuration has been provided

  on .terraform/modules/gke_zero/google/_modules/gke/provider.tf line 4, in provider "kubernetes":
   4: provider "kubernetes" {

Am I forgetting some configuration or could this be a bug?

Cheers!

• move ingress to common module
• make non nginx specific

For k8s versions below 1.16, installing cert-manager requires running kubectl apply -f with the --validate=false flag (cert-manager install docs). Opening this issue to figure out the best way to make this possible while including cert-manager in the manifests.

Hi,

When provisioning the GKE infrastructure, it would be great to be able to pull images from the GCR that is within the same project as the GKE cluster.

Currently, there is an error:

Failed to pull image "gcr.io/xxxx-xxx/img:tag": rpc error: code = Unknown desc = Error response from daemon: pull access denied for gcr.io/xxxx-xxx/api, repository does not exist or may require 'docker login'

The .github/workflows/main.yaml pipeline at the Get Started tutorial should be updated now that new repositories are named "main" instead of "master", I tried this change and it seems to be working:

github.ref == 'refs/heads/master' || github.ref == 'refs/heads/main' || startsWith(github.ref, 'refs/tags/apps-deploy-')

without this change, Terraform apply won't run in the ops cluster when merging into master

To be able to support a variety of authz use-cases users need the ability to configure the mapRoles and mapUsers sections of the aws-auth configmap.

The AWS Cluster Autoscaler needs to know the name of the cluster for auto discovery. Currently the recommended way of getting this information in the deployment is through secrets. Only the cluster name is needed, however, any other information may become beneficial later on.

Hi,

I've been reading about kubestacks and I like the idea of gitops though we are using TF Cloud and bootstrap documentation focuses only on running TF locally or via pipeline. Is it possible to implement kuberstack on TF Cloud? Thank you

We aim to support both separate project_id for ops and apps as well as same project_id for ops and apps so that teams that can't easily have a second project_id can use Kubestack.

For that, ops and apps should not use the default compute service account for nodes anymore but have dedicated, per cluster service accounts.

Step #3 - "terraform destroy": Error: Error applying plan:
Step #3 - "terraform destroy": 
Step #3 - "terraform destroy": 3 error(s) occurred:
Step #3 - "terraform destroy": 
Step #3 - "terraform destroy": * module.eks_zero.module.cluster.module.node_pool.var.cluster_ca: Resource 'aws_eks_cluster.current' does not have attribute 'certificate_authority.0.data' for variable 'aws_eks_cluster.current.certificate_authority.0.data'
Step #3 - "terraform destroy": * module.eks_zero.module.cluster.module.node_pool.var.cluster_endpoint: Resource 'aws_eks_cluster.current' does not have attribute 'endpoint' for variable 'aws_eks_cluster.current.endpoint'
Step #3 - "terraform destroy": * module.eks_zero.module.cluster.module.node_pool.var.cluster_name: Resource 'aws_eks_cluster.current' does not have attribute 'name' for variable 'aws_eks_cluster.current.name'

Possibly related: terraform-aws-modules/terraform-aws-eks#262

There are some instances where the networking setup requires different approaches to what KubeStack has (for instance running the cluster using Cloudflare as the DNS). it would be great to pull out the Ingress setup and make it more modular so you could have an opt-In approach.

To support the common setup of one org for identities and sub orgs for the clusters, aws-iam-authenticator needs to be configured to assume a role. The role may need to be created first. And we need to figure out if and if so how to allow assuming the created role.

Automating the creation and setup of that role seems preferable over expecting the user to provide a role. Subject to investigation and discussion.

We aim to support both separate project_id for ops and apps as well as same project_id for ops and apps so that teams that can't easily have a second project_id can use Kubestack.

For that, ops and apps should not use the default VPC for nodes anymore but have dedicated, per cluster VPCs.

The AKS module currently creates a random password to use for the service principal valid for 90 days. However the password is currently not rotated. There are instructions how to do this using the Azure CLI.

Test if the credential rotation can be automated using Terraform and local provisioners. So that we can keep a password that's only valid for a limited number of days and then rotate it before it expires.

Problem

I have a need to set a previous cluster version of EKS

Solutions

The solutions should be as simple as exposing the value on the resource However I am not sure how you would be able to leave it blank if you wanted to just use the latest version

Currently, the way how a kubeconfig file is generated for EKS, AKS and GKE has the downside of credentials ending up in TF state and log output from TF runs.

The idea is to instead of generating kubeconfig files, using environment variables to configure kubectl to apply the kustomize output.

The TF kubernetes provider may cause similar problems. Needs to be investigated as well.

Needs to be in the final container. We should consider to add curl too. The container is 900mb anyway already with all those Go binaries.

https://github.com/kbst/terraform-kubestack/blob/master/quickstart/src/ci-cd/Dockerfile#L76

Upstream issue: Azure/AKS#287

After manually enabling control plane logging on a Kubestack-managed cluster via the EKS console, a terraform plan shows this diff:

  # module.eks_platform.module.cluster.aws_eks_cluster.current will be updated in-place
  ~ resource "aws_eks_cluster" "current" {
        [...]
      ~ enabled_cluster_log_types = [
          - "api",
          - "audit",
          - "authenticator",
          - "controllerManager",
          - "scheduler",
        ]

It would be great if the github.com/kbst/terraform-kubestack//aws/cluster module could accept arguments to set enabled_cluster_log_types.

The current documentation states that to get the required DNS information, you need to run terraform output --module.... However, when running this with Terraform 0.12+ you get the following error:

Error: Unsupported option

The -module option is no longer supported since Terraform 0.12, because now
only root outputs are persisted in the state.

The Terraform documentation states that you must now specify outputs by adding:

output "fqdn" {
  value = module.cluster_metadata.fqdn
}

and then access it via terraform output fqdn.

Another output for ingress_zone_name_servers is also needed.

While the Ops/Apps names are valid and I understand the theory behind it. Some companies have very strict naming patterns to identify between production and non-production (also some companies are pedantic for instance all env's must have 3 letters, prd, stg). It would be great if you could switch out the default workspace names in order to allow for companies with strict naming patterns tp not move away from their best practices

Problem

We are recreating our cluster to enable the Private Node Pools, the issue seems to be that because we are recreating the cluster, the Kustomize Provider is trying to communicate with the Kubernetes Cluster on the default Localhost

Logs

Error: ResourceDiff: Get "http://localhost/api?timeout=32s": dial tcp 127.0.0.1:80: connect: connection refused

  on .terraform/modules/gke_zero/common/cluster_services/main.tf line 16, in resource "kustomization_resource" "current":
  16: resource "kustomization_resource" "current" {


Error: Process completed with exit code 1.

Steps To Reproduce

Create a cluster with the setting:

enable_private_nodes = false

Then Once created change the value:

enable_private_nodes = true

and run on that TF workspace:

terraform plan

Workaround

Currently there is a workaround by using:

terraform apply --target=<cluster module>

This will update the cluster which should then fix the problem

Are there plans on KubeStack supporting deployment on OpenStack as well? I would be very much interested in this.

Thanks!

On the infrastructure side, ops inherits everything from apps and can overwrite attributes if necessary.

On the cluster manifests side however, the both overlay breaks that same logic. Instead of ops and apps both inheriting from both, ops should simply inherit from apps and apps would inherit from common.

I was looking for a way to manage my local kind based k8s cluster with installed required components. I have initially automated the requirement; however, I looked for a better approach and learned about the Kubestack project.

I did try the Kind Kubestack example, and it worked fine; however, getting it worked without using a container is a bit hacky(compiling and installing in the terraform modules in the local .terraform folder). I thought it would be great if we will GitHub template project with kind and kustomize terraform modules with one terraform stack instead of two, and based on that, the user can easily create local or other cloud-based projects as per the requirement.

Let me know If you are up with this, and I will be happy to contribute to the corresponding changes. Thanks.

Adding node_labels to the default node pool forces a destroy and recreate plan.

  # module.aks_zero.module.cluster.azurerm_kubernetes_cluster.current must be replaced
-/+ resource "azurerm_kubernetes_cluster" "current" {
     [...]

      ~ default_node_pool {
          - availability_zones    = [] -> null
          - enable_node_public_ip = false -> null
          ~ max_pods              = 110 -> (known after apply)
            name                  = "default"
          ~ node_count            = 1 -> (known after apply)
          ~ node_labels           = { # forces replacement
              + "kubestack.com-cluster_domain"          = "azure.infra.serverwolken.de"
              + "kubestack.com-cluster_fqdn"            = "kbstacctest-ops-westeurope.azure.infra.serverwolken.de"
              + "kubestack.com-cluster_name"            = "kbstacctest-ops-westeurope"
              + "kubestack.com-cluster_provider_name"   = "azure"
              + "kubestack.com-cluster_provider_region" = "westeurope"
              + "kubestack.com-cluster_workspace"       = "ops"
            }
          - node_taints           = [] -> null
          ~ orchestrator_version  = "1.18.14" -> (known after apply)
          - tags                  = {} -> null
            # (7 unchanged attributes hidden)
        }
    }

Problem

The service account name that is created, has a limit of 28 characters, this means that if your workspace name is to long i.e (kubestack-staging) when you run apply you will get the following error:

Acquiring state lock. This may take a few moments...
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.
module.gke_zero.module.cluster.data.external.gcloud_account: Refreshing state...
module.gke_zero.module.cluster.data.google_client_config.default: Refreshing state...
module.gke_zero.module.cluster.module.cluster_services.data.kustomization.current: Refreshing state...
------------------------------------------------------------------------
Error: "account_id" ("sp-kubestack-staging-us-central1") doesn't match regexp "^[a-z](?:[-a-z0-9]{4,28}[a-z0-9])$"
  on .terraform/modules/gke_zero/google/_modules/gke/service_account.tf line 1, in resource "google_service_account" "current":
   1: resource "google_service_account" "current" {

Short term workaround

Obviously the work around is to minimize the workspace name, however if this is the proposed solution organisations might have to change naming standards (i.e a organisation that is using staging everywhere would need to start using stg). This should then be documented

Proposed fix

I am not sure about the complexity that this would create, but a potential fix would be to remove the region that is added as the suffix thus allowing for more flexibility in naming conventions

Hi, really liking Kubestack so far! 🙏

It would be useful if the kubestack/framework image had a location where arbitrary scripts could be placed, which would then be executed at runtime by the default entrypoint.

This would facilitate running some extra tasks at runtime without needing to override the default entrypoint (and fixing it when that entrypoint changes).

It would enable some nice and lazy thing like, say, authenticating to a cluster in CI (just to illustrate):

FROM kubestack/framework:v0.11.0-beta.0-eks
RUN echo "[ -f $HOME/.kube/config ] || aws eks update-kubeconfig --name platform-apps-us-east-2 --alias woot --region us-east-2" >> /opt/bin/entrypoint.d/cluster-auth.sh \
    && chmod +x /opt/bin/entrypoint.d/cluster-auth.sh

Example of other images using this pattern

• postgres uses /docker-entrypoint-initdb.d/ as described here
• nginx uses /docker-entrypoint.d/ as shown here

Following upstream, the new default for GKE clusters is to use private nodes. Private nodes do not have internet access and require a NAT gateway.

Check if upstream includes a NAT gateway by default, if so, include one as well and add opt-out varialbe for NAT gateway provisioning.

Replace the external data source currently used for aws_iam_authenticator integration with the eks_cluster_auth data source.

https://www.terraform.io/docs/providers/aws/d/eks_cluster_auth.html

To enable cluster-autoscaler on EKS, you need the AmazonEC2ContainerServiceAutoscaleRole added to the worker nodes.

Filed upstream issue: hashicorp/terraform-provider-azurerm#7716

We need to use the same user id inside the container, as the user has outside the container, so that files written to the mounted volume from inside the container are accessible by the user on the host.

In addition to that the nss-wrapper script used as the ENTRYPOINT ensures a user and group are defined inside the container with matching user and group ids.

This works well in the current implementation for Mac and Linux where the user most likely runs as an unprivileged user. On Windows using WSL, it seems the user is always root. For some reason uid 0 and gid 0 outside the container breaks the script when the container is started like this:

docker run --rm -ti \
    -v `pwd`:/infra \
    -u `id -u`:`id -g` \
    IMAGE:TAG

Instead of trial and error it would be good to have a section in the documentation with the minimum required roles

It seems EKS is now creating the aws-auth configmap, which leads to the TF run failing when the module tries to create it. This is probably related to the change to the node_group resource. 7885080

https://cloud.google.com/kubernetes-engine/docs/concepts/security-overview#control_plane_security