Atm, the Google service account ids are prefixed with the cluster name. The total length is 30 characters. This can easily lead to problems:

Error: "account_id" ("pr-6999-252-terraform-boot-vault-ko") doesn't match regexp "^[a-z](?:[-a-z0-9]{4,28}[a-z0-9])$"
  on ../terraform-google-**/modules/cluster/serviceaccount.tf line 28, in resource "google_service_account" "kaniko_sa":
  28: resource "google_service_account" "kaniko_sa" {

Let's try to ensure that the name is capped in case it is too long.

At the moment cluster creation and testing happens in the same pipeline step within the ci.sh script. When looking at the logs it would be nice to have separate steps for that.

The reason for having the tests in ci.sh is that the cluster destruction happens currently as well in ci.sh via a tap handler.

One solution would be to split ci.sh into two scripts.

When setting up a gke cluster using this module (v1.4.0) with a non-default zone (europe-west1-b in my case) the terraform module does not copy it into the resulting jx-requirements.yml.

Therefore calling jx boot -r jx-requirements.yml afterwards fails since jx-requirements.yml points to zone us-central1-a.

Currently, the example uses:

module "jx" {
  source  = "jenkins-x/jx/google"

  gcp_project = var.gcp_project
}

That's confusing since using a variable for the GCP project would also force you to define a variables.tf. To simplify the example should not use variables, but show to use the property inlined.

I am logged in with correct email on gcloud but I get all this errors:

Steps to reproduce:

• gcloud init -> choose email and project by id
• gcloud auth application-default login
• add main.tf ->

  module "jx" {
    source = "jenkins-x/jx/google"
    gcp_project = "[project-name]"
    cluster_name = "[cluster-name]"
    force_destroy = true
    version = "1.3.2"
    zone = "europe-west3-a"
  }
  

• terraform init
• terraform apply

Expected:
Finish apply process successfully

When trying to execute terraform destroy, the following error message pops up.

Error: Error reading Storage Bucket "backup-jx-demo-4cabfed7eb9f": googleapi: Error 403: [email protected] does not have storage.buckets.get access to backup-jx-demo-4cabfed7eb9f., forbidden

When I execute terraform plan, there are 54 resources to be created. Are all those mandatory? Also, what are they used for? If terraform plan is also used as "executable specification of requirements", we need to know what is each of those resources used for, so that we know what to create depending on the choices of the components we want to use with JX.

It seems I cannot specify the location of the bucket for the backups: https://github.com/jenkins-x/terraform-google-jx/blob/master/modules/backup/main.tf#L7

I do not see an option. And when I plan I get the following:

# module.jx.module.backup.google_storage_bucket.backup_bucket will be created
  + resource "google_storage_bucket" "backup_bucket" {
      + bucket_policy_only = (known after apply)
      + force_destroy      = false
      + id                 = (known after apply)
      + location           = "US"
      + name               = (known after apply)
      + project            = (known after apply)
      + self_link          = (known after apply)
      + storage_class      = "STANDARD"
      + url                = (known after apply)
    }

      + location           = "US"

As EU citizen, I generally want to store my data in the EU, or at least have the option to. See: https://cloud.google.com/storage/docs/locations

After successfully creating a cluster as per Readme, the resulting jx-requirements.yaml file does not have a properly formed empty value for devEnvApprovers.

cluster:
  clusterName: "dks-cjxd8-terra"
  devEnvApprovers: 

  environmentGitOwner: ""
  project: "apps-dev-229310"
  provider: gke
  zone: "us-central1-a"

To better control, semantic versioning lets, for now, remove the release pipeline config and to only manual tags. For Terraform Module releases a tag is all that is needed.

With the current release pipeline, we always will get a patch version increment which is not what you want if you want to increase minor or patch.

The current approach generates a jx-requirements.yml which is only useful for the initial jx boot. After the initial jx boot the generated jx-requirements.yml is disconnected from the jx-requirements.yml in the boot repo.

See also #72

The intention of the jx-requirements.yaml file is that it's only ever read at the initial setup of Jenkins X on a new cluster. With that said, producing the local_file resource means that terraform constantly manages that resource and wants to update a file which no longer has as much relevance to the cluster than it did.
The proposal is to define the jx-requirements.yaml as an output that could possibly be used to perform the initial boot like (after the terraform module has created all of the resources):

jx boot -r <(terraform output jx-requirements)

Having the Jenkins X requirements available in this manner removes the pain of having to maintain the local file within terraform and the git repository.

Certain flags enable conditional resources. We should be able to tests that these conditional work by using shellspec and the terraform plan. Looking at the plan should be enough to assert that certain resources will or will not be created.

This is to match the expected default for jx boot and having to avoid using the -r option.

Initially the tf module generates the jx-requirements.yml which is meant to be provided to jx boot -r.
However jx boot creates a new environment git repository that contains jx-requirements.yml as well.
This is potentially inconsistent because changes made using subsequent tf module executions do not make it into the environment repository.

An approach to solve this could be to let the tf module write the information contained within jx-requirements.yml into the k8s cluster so that jx boot could read it from there more consistently.

The latest jx version allows using an internal (using the Vault operator) or external Vault instance for secret storage. The Terraform module should allow configuring either of these two options.

main.tf:

module "jx" {
  source  = "jenkins-x/jx/google"
  version = "1.2.1"
  gcp_project = "foo"
}

$ terraform init 
$ terraform plan
Error: Error in function call

  on .terraform/modules/jx/terraform-google-jx-1.2.1/main.tf line 186, in resource "local_file" "jx-requirements":
 186:   content = templatefile("${path.cwd}/modules/jx-requirements.yaml.tpl", {

After running terraform apply using terraform file

module "jx" {
  source  = "jenkins-x/jx/google"

  gcp_project = "<my-gcp-project-id>"
}

At the end of process I get:

Error: Error applying IAM policy for service account 'projects/x-project-275408/serviceAccounts/[email protected]': Error setting IAM policy for service account 'projects/x-project-275408/serviceAccounts/[email protected]': googleapi: Error 400: Identity namespace does not exist (x-project-275408.svc.id.goog)., badRequest

on .terraform/modules/jx/terraform-google-jx-1.2.5/modules/backup/main.tf line 44, in resource "google_service_account_iam_member" "velero_sa_workload_identity_user":
44: resource "google_service_account_iam_member" "velero_sa_workload_identity_user" {

Error: Error applying IAM policy for service account 'projects/x-project-275408/serviceAccounts/[email protected]': Error setting IAM policy for service account 'projects/x-project-275408/serviceAccounts/[email protected]': googleapi: Error 400: Identity namespace does not exist (x-project-275408.svc.id.goog)., badRequest

on .terraform/modules/jx/terraform-google-jx-1.2.5/modules/cluster/serviceaccount.tf line 73, in resource "google_service_account_iam_member" "build_controller_sa_workload_identity_user":
73: resource "google_service_account_iam_member" "build_controller_sa_workload_identity_user" {

Error: Error applying IAM policy for service account 'projects/x-project-275408/serviceAccounts/[email protected]': Error setting IAM policy for service account 'projects/x-project-275408/serviceAccounts/[email protected]': googleapi: Error 400: Identity namespace does not exist (x-project-275408.svc.id.goog)., badRequest

on .terraform/modules/jx/terraform-google-jx-1.2.5/modules/cluster/serviceaccount.tf line 124, in resource "google_service_account_iam_member" "kaniko_sa_workload_identity_user":
124: resource "google_service_account_iam_member" "kaniko_sa_workload_identity_user" {

Error: Error applying IAM policy for service account 'projects/x-project-275408/serviceAccounts/tf-jx-arriving-frog-tekton@x-project-275408.iam.gserviceaccount.com': Error setting IAM policy for service account 'projects/x-project-275408/serviceAccounts/tf-jx-arriving-frog-tekton@x-project-275408.iam.gserviceaccount.com': googleapi: Error 400: Identity namespace does not exist (x-project-275408.svc.id.goog)., badRequest

on .terraform/modules/jx/terraform-google-jx-1.2.5/modules/cluster/serviceaccount.tf line 133, in resource "google_service_account_iam_member" "tekton_sa_workload_identity_user":
133: resource "google_service_account_iam_member" "tekton_sa_workload_identity_user" {

See #74

Hitting the same issue with version 1.3.2

resource "kubernetes_namespace" "cert-manager" {
  metadata {
    name = var.cert-manager-namespace
  }
  lifecycle {
    ignore_changes = [
      metadata[0].labels,
      metadata[0].annotations,
    ]
  }
}

This module creates namespaces (jx, cert-manager, velero), which seems to be outside the scope of this modules intent. From past experiences it becomes difficult when mixing resource creation where resources have differing lifecycles and where more than one agent is responsible for configuring said resources (can cause conflicts).

The idea is to use the same approach as for jx create cluster.
See https://github.com/jenkins-x/terraform-aws-eks-jx/blob/master/local.tf#L10

The tls_email variable needs to be conditionally set if custom domains are enabled. Right now when you set parent_domain there is no validation which ensures that you also set tls_email. Missing to set this variable will read to issues later during jx boot.

Using Terraforms variable validation - https://www.terraform.io/docs/configuration/variables.html - this conditional dependency could be expressed. On the downside, variable validation is atm still an experimental feature.

See https://cloud.google.com/kubernetes-engine/docs/how-to/hardening-your-cluster#restrict_network_access_to_the_control_plane_and_nodes

We could optionally give the ability to restrain access to k8s control plane to only a specific set of CIDR blocks, by specifying https://www.terraform.io/docs/providers/google/r/container_cluster.html#master_authorized_networks_config

There doesn't exist to be a method to specify a version of the kubernetes or release channel for GKE clusters in this module. Allowing the user to specify this would be beneficial.

ATM, the backup configuration (backup storage bucket, namespace, service account, etc) are always created. Given that this is not fully tested/supported yet, it makes sense to only create resources conditionally. One needs to explicitly opt in.

On the downside, it means that if you at a later stage want to enable it, you can create the missing resources by enabling the feature and re-running terraform apply, but then you will have to manually sync the required jx-requirements.yml changes into the jx-requirements.yml of your dev repository.

Even if a parent_domain parameter is not specified in a main.tf file, include a disabled ingress section in the jx-requirements.yaml output. This will better allow for the subsequent ability to supply DNS and/or TLS values prior to running jx boot

As per https://www.terraform.io/docs/configuration/modules.html#provider-version-constraints-in-modules

The current repo layout looks like this:

├── jx
│   ├── main.tf
│   ├── modules
│   │   ├── backup
│   │   ├── cluster
│   │   ├── dns
│   │   └── vault
│   ├── output.tf
│   └── variables.tf
├── jx-requirements.yaml
├── jx-requirements.yaml.tpl
├── main.tf
├── output.tf
└── variables.tf

The intermediate jx directory seems superfluous and should be removed. This means we are effectively merging main.tf of the root level with the one in jx.

As a side effect, we have to duplicate/copy fewer variables and outputs across module boundaries.

Currently uses the default of pd-standard. In my instance I wanted to specify pd-ssd.

To allow to apply custom labels to the created cluster, it makes sense to allow specifying cluster resource labels as input variables to the main Terraform script.

They belong to the google_container_cluster resource and look like this:

        resource_labels  = {
            "env"       = "staging"
            "managedby" = "terraform"
        }

This could bee achieved via the Helm provisioner. It would require further work and refactoring on the Jenkins X side, but eventually, it should be possible.

See also https://github.com/jenkins-x-labs/jenkins-x-installer

See https://github.com/jenkins-x/terraform-aws-eks-jx/blob/master/modules/cluster/irsa.tf#L283

The UI when installed needs permissions to could storage for accessing the logs

At the moment storage buckets are created within the region "US" and ignoring the zone variable, since the location is not set via script.

We downgraded version to 2.12.0 as part of issue #40. There was an underlying issue in the provider which was captured in this issue hashicorp/terraform-provider-google#4276.

Looks like the issue is not closed and resolved, so we can try to upgrade the provider again, especially since we are atm using a very old version.

Currently, we are not creating the managed zone with Terraform, but require it to be created upfront when using a custom domain. There are two reasons for that:

1. If Terraform creates the managed zone and Jenkins X get installed, new recordsets are getting created in the zone. This prevents a clean destroy since you first have to manually delete all records
2. More importantly, if one creates the managed zone as part of the script, one will get a set of nameservers as output which one has to manually set with the DNS registrar. You then have to wait until the DNS settings have propagated before running jx boot.

Making the managed zone creation optional might be useful in some cases where you want to manage as much as possible via Terraform.

Using min_node_count and max_node_count

Using the min_node_count and max_node_count 
module "jx" {
  source  = "jenkins-x/jx/google"
  gcp_project = ...

  cluster_name = ...
  git_owner_requirement_repos = "hemp0r"
  min_node_count = 1
  max_node_count = 2
  dev_env_approvers = ["hemp0r"]
  zone = "europe-west3"
  
  force_destroy = true 
}

leads to

+ resource "google_container_node_pool" "jx_node_pool" {
      + cluster             = ...
      + id                  = (known after apply)
      + initial_node_count  = 3
      + instance_group_urls = (known after apply)
      + location            = "europe-west3"
      + max_pods_per_node   = (known after apply)
      + name                = "autoscale-pool"
      + name_prefix         = (known after apply)
      + node_count          = (known after apply)
      + project             = (known after apply)
      + region              = (known after apply)
      + version             = (known after apply)
      + zone                = (known after apply)

      + autoscaling {
          + max_node_count = 5
          + min_node_count = 3
        }

      + management {
          + auto_repair  = true
          + auto_upgrade = false
        }

      + node_config {
          + disk_size_gb      = 100
          + disk_type         = (known after apply)
          + guest_accelerator = (known after apply)
          + image_type        = (known after apply)
          + labels            = (known after apply)
          + local_ssd_count   = (known after apply)
          + machine_type      = "n1-standard-2"
          + metadata          = (known after apply)
          + oauth_scopes      = [
              + "https://www.googleapis.com/auth/cloud-platform",
              + "https://www.googleapis.com/auth/compute",
              + "https://www.googleapis.com/auth/devstorage.full_control",
              + "https://www.googleapis.com/auth/logging.write",
              + "https://www.googleapis.com/auth/monitoring",
              + "https://www.googleapis.com/auth/service.management",
              + "https://www.googleapis.com/auth/servicecontrol",
            ]
          + preemptible       = false
          + service_account   = (known after apply)

          + workload_metadata_config {
              + node_metadata = "GKE_METADATA_SERVER"
            }
        }
    }

Would it make sense to also support Azure? Looks like it would be based on community feedback jenkins-x/jx#6780 (comment)

/cc @daveconde

I'm getting the following error message.

Error: Error applying IAM policy for service account 'projects/jx-20200617185112/serviceAccounts/[email protected]': Error setting IAM policy for service account 'projects/jx-20200617185112/serviceAccounts/[email protected]': googleapi: Error 400: Identity namespace does not exist (jx-20200617185112.svc.id.goog)., badRequest

I tried it twice on a fresh Google project with the same result. However, if I re-run terraform apply again, it works. I'm guessing its some kind of a race condition and it tries to create resources before its dependencies are created.

See. https://www.terraform.io/docs/providers/google/guides/provider_reference.html

The zone field should be changed to location. Depending on the value, GKE will be zonal (e.g., us-east1-b) or regional (us-east1).

Make sure to update the docs to clarify that the initial number of nodes is multiplied by the number of zones. If, for example, a cluster is regional, 1 node size would result in 3 nodes (one in each of the three zones).

We need to decide how we want to deal with state management? At the very least we need to document the importance of remote state storage with some references on how to do it. Even better might be to provide examples.

The helper script release.sh uses:

jx step changelog -v $version -p $prev_tag_base -r $current_tag_base --generate-yaml=false --no-dev-release --update-release=false

This works, however, this step is supposed to run in cluster and prints a lot of warnings when running locally. Partly because it will always use the pipeline's user credentials.
Also, due to the credential issue, the changelog is just written to the console and one needs to manually cut&paste.

A short jq script should do the trick.

Detailed by the issue here: hashicorp/terraform-provider-google#4276

Get bad request errors like so:

  on jx/modules/cluster/serviceaccount.tf line 60, in resource "google_project_iam_member" "tekton_sa_project_viewer_binding":
  60: resource "google_project_iam_member" "tekton_sa_project_viewer_binding" {

The resolution is to modify jx/main.tf and change:

provider "google" {
  version = "~> 3.10.0"
  project = var.gcp_project
  zone    = var.zone
}

provider "google-beta" {
  version = "~> 3.10.0"
  project = var.gcp_project
  zone    = var.zone
}

to

provider "google" {
  version = "~> 2.12.0"
  project = var.gcp_project
  zone    = var.zone
}

provider "google-beta" {
  version = "~> 2.12.0"
  project = var.gcp_project
  zone    = var.zone
}

terraform destroy produced the following output.

...
module.jx.module.cluster.google_container_cluster.jx_cluster: Destruction complete after 2m47s

Error: Error trying to delete bucket vault-jx-demo-53af71c741fa containing objects without `force_destroy` set to true



Error: Error trying to delete bucket logs-jx-demo-53af71c741fa containing objects without `force_destroy` set to true

That part is OK.

After that, I added force_destroy = true' to my Terraform definition and re-run terraform destroy`. It produced the following error.

...

Error: Error trying to delete bucket vault-jx-demo-53af71c741fa containing objects without `force_destroy` set to true



Error: Error trying to delete bucket logs-jx-demo-53af71c741fa containing objects without `force_destroy` set to true



Error: Invalid index

  on .terraform/modules/jx/terraform-google-jx-1.4.0/modules/cluster/outputs.tf line 22, in output "report_storage_url":
  22:     value = google_storage_bucket.report_bucket[0].url
    |----------------
    | google_storage_bucket.report_bucket is empty tuple

The given key does not identify an element in this collection value.


Error: Invalid index

  on .terraform/modules/jx/terraform-google-jx-1.4.0/modules/cluster/outputs.tf line 26, in output "repository_storage_url":
  26:     value = google_storage_bucket.repository_bucket[0].url
    |----------------
    | google_storage_bucket.repository_bucket is empty tuple

The given key does not identify an element in this collection value.

We've experienced an issue within our repository where we use a newer 1.4.0 version of the local terraform provider. Due to changes in the minor version bumps the current constraint of ~> 1.2.0 of the local provider corrupts our terraform state by adding content_base64 = null, resulting in the following error:

Error: unsupported attribute "content_base64"

Going forward I think all provider scopes in this module need to have their version constraints relaxed (whilst retaining the minimum version constraint), thus allowing the consuming project to determine which version to use.
https://www.terraform.io/docs/configuration/modules.html#provider-version-constraints-in-modules

When the tf module is run without parent_domain configured tf does not create the cert-manager namespace. However if one decides to configure parent_domain and reapply the module afterwards it fails with the following error since the namespace already exists:

Error: namespaces "cert-manager" already exists
  on .terraform/modules/jx/modules/dns/main.tf line 106, in resource "kubernetes_namespace" "cert-manager":
 106: resource "kubernetes_namespace" "cert-manager" {

As a fix the cert-manager namespace could always be created by tf.

module version: 1.4.0
google provider version: 3.25.0
Terraform v0.12.24
OS Ubuntu 20.04
Linux Kernel: 5.4

When specifying the service account credentials file like this:

provider "google" {
  credentials = file("secret.json")
  project = "my-project"
  zone = "europe-west1-b"
}

terraform plan fails with:

Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.


Error: google: could not find default credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information.

  on .terraform/modules/automation-cluster/terraform-google-jx-1.4.0/main.tf line 13, in provider "google":
  13: provider "google" {

The very same configuration work e.g. with (from the gcp website):

// Terraform plugin for creating random ids
resource "random_id" "instance_id" {
 byte_length = 8
}

// A single Google Cloud Engine instance
resource "google_compute_instance" "default" {
 name         = "flask-vm-${random_id.instance_id.hex}"
 machine_type = "f1-micro"
 zone         = "us-west1-a"

 boot_disk {
   initialize_params {
     image = "debian-cloud/debian-9"
   }
 }

// Make sure flask is installed on all new instances for later steps
 metadata_startup_script = "sudo apt-get update; sudo apt-get install -yq build-essential python-pip rsync; pip install flask"

 network_interface {
   network = "default"

   access_config {
     // Include this section to give the VM an external ip address
   }
 }
}

Workaround: Remove the credentials variable from the configuration and set the GOOGLE_CLOUD_KEYFILE_JSON environment variable.

I'm getting the following error when running the module against a newly created Google project.

...
Error: googleapi: Error 403: Kubernetes Engine API is not enabled for this project. Please ensure it is enabled in Google Cloud Console and try again: visit https://console.cloud.google.com/apis/api/container.googleapis.com/overview?project=jx-20200617144145 to do so., forbidden

  on .terraform/modules/jx/terraform-google-jx-1.4.0/modules/cluster/main.tf line 6, in resource "google_container_cluster" "jx_cluster":
   6: resource "google_container_cluster" "jx_cluster" {
...

I believe that the module is already enabling quite a few APIs so it probably makes sense to enable that one through the module as well. Otherwise, maybe we should add a note to the docs that the "Kubernetes Engine API" should be enabled (e.g., gcloud services enable container.googleapis.com --project $PROJECT_ID).

I believe this should be possible to add, but as of now, we always use the default version, and it'd be good to allow that to be overridden.

Relates to jenkins-x/jx#6957