In most (probably all) cases the Gateway API will be used with an ACME issuer because usually this Gateway will be how the services are exposed publicly. Also it probably makes more sense to test this with an ACME issuer with HTTP-01 challenge because that way we can also test that the Gateway controller configuration works.

So this issuer spec could be defaulted to ACME with Let'sEncrypt dev issuer similar to this. We can make the spec configurable so users can pass their own if they want another issuer type.

Come from #31 (comment)

At the moment users can configure the path of kubeconfig.

It would also be good if other options could be made configurable:

• cert-manager version to install
• Vault version to install
• root domain of the Vault PKI
• flags to pass to cert-manager components
• whether ClusterIssuer or an Issuer (in a particular namespace) gets created
  ...

Initial cert-manager + Vault setup, to be refactored in further PRs

(This will likely be mostly the contents of https://github.com/lonelyCZ/terraform-cm/tree/master/cm-vault)

Add the ACME issuer for cm-ingress.

The Terraform files should deploy cert-manager and a lightweight Prometheus installation with an optional Grafana(and some static cert-manager dashboard) with Prometheus configured to pull cert-manager’s metrics via cert-manager’s ServiceMonitor

Currently cert-manager supports 3 authentication methods for Vault issuers https://cert-manager.io/docs/configuration/vault/#authenticating

It would be good if users of the cert-manager +Vault workflow supported all three (so users of the workflow can either choose to create an Issuer/ClusterIssuer with either method or even all three.

The port-forward command runs in background, so by default it does not output any error messages, which makes it difficult to debug it if it fails.
Perhaps it is possible to write the ouput to a temporary file?

The Readme should indicate the purpose of the project and how to use it

We should also link the design document in the Readme https://docs.google.com/document/d/14oJux-d-91Do3DLi5eRG-wUEE4R3Hh7D4u0Yfje3auw/edit#heading=h.9chr39ggpwbe

The test that was added in #2 should run on every PR. We could do that via Github actions.

It may have been good to add to the Readme how a developer should add a new workflow.

It would be good if users had some way to pass Helm chart values via Terraform variables.
It will probably require some Terraform input variable of list or map type (maybe there is a better solution?). We might need to loop over that value to create the necessary set blocks

Note: later when we add Gateway API workflow we should be able to use the same mechanism to configure the shared cert-manager installation to work with Gateway API.
(For context for cert-manager to work with Gateway API resources, the feature gate ExperimentalGatewayAPISupport needs to be enabled by passing --feature-gates=ExperimentalGatewayAPISupport=true to cert-manager controller. When using Helm chart this can be done via extraArgs Helm value)

As this repo will mostly contain Terraform files, we could have a basic test that runs Terraform linter.

We should wait for Certificate and ClusterIssuer to become ready before considering cm-config module apply complete so that users can easier see if there are any issues.
We should be able to use the wait block to wait for Ready condition on these resources https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/resources/manifest#using-wait-to-block-create-and-update-calls

We should have a workflow for setting up cert-manager with a ingress controller to test cert issuance via ingress shim. We could use Ingress NGINX for testing this.

The workflow should deploy

• cert-manager
• ingress NGINX controller
• a cert-manager issuer, preferably ACME LetsEncrypt
• an Ingress resource with cert-manager annotations to select the deployed issuer

Here is how we set up Ingress NGINX in cert-manager e2e tests. I think we should be able to use the latest ingress NGINX version.
Let me know if you think there will be issues with testing ACME issuer- I can do testing in cloud if needed.

At the moment 4 are three separate vars files in ./cm-vault/*.

We should have a single file, so that if users want to, for example, provide a different kube config file path, they only need to specify it in one place.

(I assume that it is possible to pass a variables file to terragrunt run-all apply command)

At the moment cert-manager deployed from cm-vault workflow authenticates to Vault using token authentication method.

Debugging issues that require a specific authentication method is a common task- it would be good if it was also possible to optionally set up authentication via AppRole.

I see that the Terraform Vault provider has approle resources.

Users could select which authentication mechanism (token or approle) they want via a variable, i.e authentication_mechanism. Perhaps we could use count to make some resources get created optionally, i.e

resource "vault_auth_backend" "approle" {
  count = var.authentication_mechanism == "approle" ? 1 : 0
  type = "approle"
}

We currently have one Terragrunt workflow that creates cert-manager, Vault and some cert-manager and Vault resources.

We will be adding more workflows, starting with one that installs cert-manager and Gateway API resources and a gateway controller.

It would be great if we could re-use the functionality that installs cert-manager to reduce the maintenance burden that comes with having the same code in two places.

We could do this, for example, using Terraform modules and child modules.
As an example the repo structure could be something like

- modules
  - cm-install # terraform files from the current ./cm-vault/cm-install
  - cm-config
  - vault-install
  ...
- workflows
  - cm-vault
    - installation
      - main.tf
      - terragrunt.hcl
    - cm-config
      - main.tf
      - terragrunt.hcl
    - vault-config
      - main.tf
      - terragrunt.hcl
    ...
  - cm-gateway
    - installation
      - main.tf
   ...

where the contents of the ./workflows/cm-vault/installation/main.tf` could be something like

module {
  source = "../../../modules/cm-install"
  ...
}
module {
  source = "../../../modules/vault-install"
}

Or similar, I haven't actually tested if the above works.

The main thing that I would suggest is to try to refactor this in a way that is as much standard/known best practices as possible to make it easier for devs who are already familiar with standard repo structures to work with this repo.
It would be good if we could look up some example repository structures for both Terragrunt and Terraform modules, I see that for example Gruntworks have some in in their Github repository https://github.com/gruntwork-io?q=example&type=all&language=&sort=

Perhaps we can find a couple examples that are relevant and link here or in the associated PR.

It looks like some defaulted parameters in Vault's backend role get modified if terragrunt apply-all

Terraform will perform the following actions:

  # vault_pki_secret_backend_role.role will be updated in-place
  ~ resource "vault_pki_secret_backend_role" "role" {
        id                                 = "pki/roles/cert-manager-io"
      ~ key_usage                          = [
          - "DigitalSignature",
          - "KeyAgreement",
          - "KeyEncipherment",
        ]
        name                               = "cert-manager-io"
        # (36 unchanged attributes hidden)
    }

To reproduce:

1. Run terragrunt run-all apply from cm-vault directory
2. Run terragrunt run-all apply again
3. Observe output that suggests that vault_pki_secret_backend_role.role got modified

To prevent this from happening, we probably want to explicitly specify the key usage that Vault otherwise seems to default https://registry.terraform.io/providers/hashicorp/vault/latest/docs/resources/pki_secret_backend_role#key_usage

When I run terragrunt run-all destroy to destroy the test environment, there are remain cert-manager and vault namespace

[root@master68 testing-addons]# kubectl get ns
NAME              STATUS   AGE
cert-manager      Active   7d3h
default           Active   249d
kube-node-lease   Active   249d
kube-public       Active   249d
kube-system       Active   249d
vault             Active   7d3h

and the demo-app-tls secret that is issued by ClusterIssuer also isn't cleaned, which could affect the next test.

[root@master68 testing-addons]# kubectl get secret
NAME                                       TYPE                                  DATA   AGE
demo-app-tls                               kubernetes.io/tls                     3      7d3h

At the moment the vault addon installation only works with preconfigured port forwarding.

We should automate port forwarding and make sure that it works not just on Kind, but also other kinds of Kubernetes.

Add inital workflow to configure cert-manager with Gateway API resources.

This workflow should:

• install cert-manager configured to set the experimental Gateway API feature to true (see #24 )
• install Gateway API CRDs
• install a controller that will parse Gateway resources and any related resources. This controller could be, for example, Contour. For reference, see how we install Contour and related resources for cert-manager e2e tests. (In a later PR we could add a few more controllers, i.e Traefik
• install cert-manager issuer

#23 and #24 should be worked on first.

There will likely have to be manual steps to configure networking/DNS, depending on users environment.