I'm thinking of using vault-init container to use HachiCorpVault in GKE.
I installed vault with Helm and deployed vault-init as a sidecar container.

However, the encryption of the root token fails with the following error.

2020/04/17 06:18:28 Starting the vault-init service...
2020/04/17 06:18:28 Head http://127.0.0.1:8200/v1/sys/health: dial tcp 127.0.0.1:8200: connect: connection refused
2020/04/17 06:18:39 Vault is not initialized.
2020/04/17 06:18:39 Initializing...
2020/04/17 06:18:54 Encrypting unseal keys and the root token...
2020/04/17 06:18:54 Post https://cloudkms.googleapis.com/v1/projects/{xxx}/locations/{xxx}/keyRings/{xxx}/cryptoKeys/vault-init:encrypt?alt=json&prettyPrint=false: x509: failed to load system roots and no roots provided; open /etc/ssl/certs/ca-certificates.crt: permission denied
2020/04/17 06:18:54 Next check in 10s
....

I don't know why I'm getting a permission error in /etc/ssl/certs/ca-certificates.crt, is there a way to deal with it?

• helm chart
  • https://github.com/hashicorp/vault-helm/archive/v0.5.0.tar.gz
• vault-init
  • sethvargo/vault-init:1.1.0

Hi @sethvargo ,

Hope you are doing well!!!

I was trying to configure vault backed by consul where both should be hosted on k8s cluster and using AWS KMS to auto unseal. After searching a lot i landed up on your youtube video and github example(excellent articulation, you made my life easy to understand the whole), using which i was able to run vault in k8s cluster with auto unseal. But initial vault init needs to be done as i implemented vault statefulset without auto vault init. I was going through your GO code base of vault-init where Google storage is used as vault backend along with Google KMS for auto unseal, but since my vault cluster is backed by consul and AWS KMS, hence i was not able use your vault init AS-IS.

I need little help from your side on below:

• how this vault init is happening at backend(simple briefing about the whole workflow)
• does it feasible with consul and if yes please point me with some examples so that i can achieve the auto vault init as well.

thanks in advance,

regards

I installed vault with Helmfile and deployed vault-init as a sidecar container.

However, it keeps logging on readinessprobe I guess.

2020/10/01 15:13:38 Starting the vault-init service...
2020/10/01 15:13:38 Head https://127.0.0.1:8200/v1/sys/health: http: server gave HTTP response to HTTPS client
2020/10/01 15:13:48 Head https://127.0.0.1:8200/v1/sys/health: http: server gave HTTP response to HTTPS client
2020/10/01 15:13:58 Head https://127.0.0.1:8200/v1/sys/health: http: server gave HTTP response to HTTPS client
2020/10/01 15:14:08 Head https://127.0.0.1:8200/v1/sys/health: http: server gave HTTP response to HTTPS client
2020/10/01 15:14:18 Head https://127.0.0.1:8200/v1/sys/health: http: server gave HTTP response to HTTPS client
2020/10/01 15:14:28 Head https://127.0.0.1:8200/v1/sys/health: http: server gave HTTP response to HTTPS client
2020/10/01 15:14:38 Head https://127.0.0.1:8200/v1/sys/health: http: server gave HTTP response to HTTPS client
2020/10/01 15:14:48 Head https://127.0.0.1:8200/v1/sys/health: http: server gave HTTP response to HTTPS client
2020/10/01 15:14:58 Head https://127.0.0.1:8200/v1/sys/health: http: server gave HTTP response to HTTPS client
2020/10/01 15:15:08 Head https://127.0.0.1:8200/v1/sys/health: http: server gave HTTP response to HTTPS client
2020/10/01 15:15:18 Head https://127.0.0.1:8200/v1/sys/health: http: server gave HTTP response to HTTPS client

• helm chart
  https://github.com/hashicorp/vault-helm/archive/v0.7.0.tar.gz
• vault-init
  • sethvargo/vault-init:0.1.2
  • sethvargo/vault-init:0.1.1
  • sethvargo/vault-init:1.1.0

I tried with different versions of vault-init, but no luck.

I can not run exec /bin/sh into vault-init container which is strange after checked it is golang based image.

OCI runtime exec failed: exec failed: container_linux.go:349: starting container process caused "exec: \"/bin/sh\": stat /bin/sh: no such file or directory": unknown command terminated with exit code 126

Hi,

I would like to know, would the vault-init container support rotating keys as described here: https://cloud.google.com/kms/docs/rotating-keys or should any special care be taken when enabling key rotation?

It looks like the original code had TLS verification disabled by default and hardcoded. I assume this was due to the expected use case of talking to Vault over a shared network namespace on 127.0.0.1.

Then this changed to configurable via VAULT_INSECURE_TLS_SKIP_VERIFY and the default remained with verification disabled: c3a324b

Next, the env var was renamed to VAULT_SKIP_VERIFY to match the vault cli/lib, and the default was changed to verification enabled: 6bbe8ba

The change of the default makes sense. But it would be nice to have a way to specify the CA bundle. I haven't tried this but I imagine one could volume mount their own CA bundle over one of the expected places used by go's stdlib such as /etc/ssl/cert.pem (https://golang.org/src/crypto/x509/root_linux.go).

Possibly more ideal would be supporting the VAULT_CACERT or VAULT_CAPATH env vars.

We noticed a failure in a weekly acceptance test we run that creates a new instance of Vault, runs some tests, then tears it down. This week's test failed to setup the Vault instance. We tracked the cause down to the 0.1.1 and 0.1.2 tags on Docker Hub having been overwritten with the latest changes. The observed issues that led us here were:

1. The default skip-TLS-verification was changed from enabled to disabled as described in #9. This led to vault-init being unable to communicate with vault.
2. Vault auto-unseal was added and set to default. This meant our Vault instance (which has not switched to native auto-unseal) was not unsealed by vault-init.

We were using vault-init:0.1.1. The docker tags for versions 0.1.1 and 0.1.2 on docker hub were overwritten with a new version that had the above two changes in behavior. https://hub.docker.com/r/sethvargo/vault-init/tags

Our fix to return the original behavior:

• set VAULT_SKIP_VERIFY=true
• set VAULT_AUTO_UNSEAL=false (we'll migrate to auto unseal soon)