Vault Agent Sidecar Injector Demo
In this demo we are going to walk through a new Vault + kubernetes integration that allows application with no Vault logic built in to source secrets from Vault. This is made possible through a new tool called vault-k8s which leverages the kubernetes mutation admission webhook to intercept and argument (or change) specifically annotated pod definitions for secrets injections.
Note: These are just examples, so USE AT YOUR OWN RISK! Please only use with a demo kubernetes cluster when learning. I am assuming you have helm installed and you know how to use Kubernetes already.
Reference Links:
- Injecting Vault Secrets Into Kubernetes Pods via a Sidecar (blog)
- Agent Sidecar Injector (docs)
- Injecting Secrets into Kubernetes Pods via Vault Helm Sidecar (hands on lab)
- https://github.com/hashicorp/vault-helm
- https://github.com/hashicorp/vault-k8s
- Video: Vault + Kubernetes Sidecar Injection
- Video: Dynamic Database Credentials with Vault and Kubernetes
Clone this demo repo
git clone https://github.com/jweissig/vault-k8s-sidecar-demo
cd vault-k8s-sidecar-demo
Clone vault-helm
git clone --branch v0.3.3 https://github.com/hashicorp/vault-helm.git
Configure a GCP k8s cluster
I am going to assume you have a GCP cluster called vault-cluster up and running already. Use this command to connect to it.
gcloud container clusters get-credentials vault-cluster --zone us-central1-a --project sidecar-injection
Check to make sure you have nodes ready with kubectl get nodes. Also, in a second term window configure a kubectl get pods loop like so.
watch -t -n1 kubectl get pods
Install Vault
Next, lets install Vault onto Kubernetes (running in dev mode).
kubectl create namespace demo
kubectl config set-context --current --namespace=demo
# helm v3
helm install vault \
--set='server.dev.enabled=true' \
./vault-helm
# or helm v2
helm install --name=vault \
--set='server.dev.enabled=true' \
./vault-helm
Configure Vault
Next, lets configure Vault.
kubectl exec -ti vault-0 /bin/sh
Next, lets configure a policy that we can then attach a role to (used for accessing secrets from a Kubernetes service account).
cat <<EOF > /home/vault/app-policy.hcl
path "secret*" {
capabilities = ["read"]
}
EOF
Then, lets apply the policy.
vault policy write app /home/vault/app-policy.hcl
Next, enable the kubernetes auth method.
vault auth enable kubernetes
Next, lets configure the kubernetes auth method, so it can communicate with Kubernetes.
vault write auth/kubernetes/config \
token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
kubernetes_host=https://${KUBERNETES_PORT_443_TCP_ADDR}:443 \
kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
Next, lets connect the policy we created earlier to a myapp role. This will be used in a minute when we create our Kubernetes service account for our demo apps so they are allowed to pull down secrets from Vault.
vault write auth/kubernetes/role/myapp \
bound_service_account_names=app \
bound_service_account_namespaces=demo \
policies=app \
ttl=1h
Next, lets create some secrets.
vault kv put secret/helloworld username=foobaruser password=foobarbazpass
vault kv put secret/payment-api APIKEY=497CA26DA27E
vault kv put secret/sendmail-api APIKEY=F5863ABDB85A
You can list the secrets by running vault kv list secret.
Keys
----
helloworld
payment-api
sendmail-api
Finally, we are done. Lets exit out.
exit
Running through the demo
Here's a breakdown of what we are going to test:
app.yaml # demo app that knows nothing about vault
patch-basic-annotations.yaml # sidecar proof of concept
patch-template-annotations.yaml # now, lets use templates
patch-template-realworld.yaml # now, lets use multiple secrets
job-pre-populate-only.yaml # show how jobs work
Lets launch our demo app that knows nothing about Vault (plus create the service account).
vi app.yaml
kubectl create -f app.yaml
kubectl exec -ti app-<POD_NAME> -c app -- ls -l /vault/secrets
Next, lets view the definition to check for annotations (there should be none).
kubectl describe pod app-<POD_NAME> | less
Next, lets inject a secret via a sidecar and prove it works.
vi patch-basic-annotations.yaml
kubectl patch deployment app --patch "$(cat patch-basic-annotations.yaml)"
kubectl exec -ti app-<POD_NAME> -c app -- ls -l /vault/secrets
kubectl exec -ti app-<POD_NAME> -c app -- cat /vault/secrets/helloworld
Lets check our annotations again (we should see some).
kubectl describe pod app-<POD_NAME> | less
Next, lets use a template to format the data into something useful.
vi patch-template-annotations.yaml
kubectl patch deployment app --patch "$(cat patch-template-annotations.yaml)"
kubectl exec -ti app-<POD_NAME> -c app -- ls -l /vault/secrets
kubectl exec -ti app-<POD_NAME> -c app -- cat /vault/secrets/helloworld
Next, lets look at a real world example with multiple secrets.
vi patch-template-realworld.yaml
kubectl patch deployment app --patch "$(cat patch-template-realworld.yaml)"
kubectl exec -ti app-<POD_NAME> -c app /bin/sh
Next, lets check out how you might configure a background job.
vi job-pre-populate-only.yaml
kubectl create -f job-pre-populate-only.yaml
kubectl logs pgdump-
Clean up
Lets delete our demo app.
kubectl delete deployment app
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent