This repo hosts a MutatingAdmissionWebhook that injects environment variables, dns options and node affinity into pod containers prior to persistence of the object. Node affinity is currently limited to RequiredDuringSchedulingIgnoredDuringExecution selector terms.
The image can be used along with several Kubernetes resources to update specific settings on your pods based on a label applied to the namespace of the pod i.e.
- Apply the label to the namespace and all pods within will have the same additional configuration applied at scheduling time.
The following options are available for configuration (if existing configuration exists then the new configuration you supply will be appended, it does not replace existing configuration).
- Environment Variables
- DNS Options
- Required Node Affinity terms
- Preferred Node Affinity terms
- Tolerations
- Topology Spread Constraints
Each configuration type is optional so your configmap or values file will only include those that you want to change.
Example config map:
apiVersion: v1
kind: ConfigMap
metadata:
name: env-injector-webhook-configmap
data:
envconfig.yaml: |
tolerations:
- key: kubernetes.azure.com/scalesetpriority
effect: NoSchedule
operator: Equal
value: spot
topologyConstraints:
- maxSkew: 1
topologyKey: topology.kubernetes.io/zone
whenUnsatisfiable: ScheduleAnyway
nodeAffinityPolicy: Honor
nodeTaintsPolicy: Honor
labelSelector:
matchLabels:
app.kubernetes.io/name: test-app
matchLabelKeys:
- pod-template-hashexample values.yaml file (Helm):
environment: {}
dnsOptions: {}
requiredNodeAffinityTerms: {}
preferredNodeAffinityTerms: {}
tolerations:
- key: kubernetes.azure.com/scalesetpriority
effect: NoSchedule
operator: Equal
value: spot
topologyConstraints:
- maxSkew: 1
topologyKey: topology.kubernetes.io/zone
whenUnsatisfiable: ScheduleAnyway
nodeAffinityPolicy: Honor
nodeTaintsPolicy: Honor
labelSelector:
matchLabels:
app.kubernetes.io/name: test-app
matchLabelKeys:
- pod-template-hash
Kubernetes 1.22.0 or above with the admissionregistration.k8s.io/v1 API enabled. Verify that by the following command:
kubectl api-versions | grep admissionregistration.k8s.io/v1
The result should be:
admissionregistration.k8s.io/v1
In addition, the MutatingAdmissionWebhook and ValidatingAdmissionWebhook admission controllers should be added and listed in the correct order in the admission-control flag of kube-apiserver.
Within this repository is a Dockerfile, this should be used when there are changes made to the Golang code.
Build and push docker image
docker build --tag k8s-env-injector:latest .
docker tag k8s-env-injector <your container repository>/k8s-env-injector:<tag>
docker push <your container repository>/k8s-env-injector:<tag>
Create a signed cert/key pair and store it in a Kubernetes secret that will be consumed by env-injector deployment
./deployment/webhook-create-signed-cert.sh \
--service env-injector-webhook-svc \
--secret env-injector-webhook-certs \
--namespace default
NOTE: This creates a secret within your namespace so you need to use this namespace for the rest of the deployment steps
Patch the MutatingWebhookConfiguration by set caBundle with correct value from Kubernetes cluster
cat deployment/mutatingwebhook.yaml | \
deployment/webhook-patch-ca-bundle.sh > \
deployment/mutatingwebhook-ca-bundle.yaml
This will update the local deployment/mutatingwebhook-ca-bundle.yaml file with a new CA bundle string, make sure to check that it also has the matching namespace file before you deploy.
Deploy resources
kubectl create -f deployment/configmap.yaml
kubectl create -f deployment/deployment.yaml
kubectl create -f deployment/service.yaml
kubectl create -f deployment/mutatingwebhook-ca-bundle.yaml
The environment inject webhook should be running now in your namespace, you can verify by:
kubectl get pods
NAME READY STATUS RESTARTS AGE
env-injector-webhook-deployment-bbb689d69-882dd 1/1 Running 0 5m
kubectl get deployment
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
env-injector-webhook-deployment 1 1 1 1 5m
Add a label to the namespace that you want env-injector to make changes to with: env-injector=enabled
kubectl label namespace default hmcts.github.com/envInjector=enabled
kubectl get namespace -L hmcts.github.com/envInjector
Output:
NAME STATUS AGE ENVINJECTOR
default Active 4d3h enabled
kube-node-lease Active 4d3h
kube-public Active 4d3h
kube-system Active 4d3h
Create a test deployment in your namespace, the following is an example that can be used for quick results:
cat <<EOF | kubectl create -f -
apiVersion: apps/v1
kind: Deployment
metadata:
name: sleep
spec:
replicas: 1
selector:
matchLabels:
app: sleep
template:
metadata:
labels:
app: sleep
spec:
containers:
- name: sleep
image: hmctspublic.azurecr.io/docker-curl
command: ["sleep","1d"]
imagePullPolicy: Always
EOF
To verify that the changes have been applied you can check the output of the pod. The following example is based on using the deployment above:
kubectl get pod -l app=sleep -o json
You should be able to see your additional config listed as part of the pod spec. If you have JQ installed you can narrow down the results to what you want to see:
kubectl get pod -l app=sleep -o json | jq '.spec.dnsOptions'
A Helm chart is also available, see env-injector-webhook. This can be installed in a single step using helm 2 or 3, e.g.
$ helm upgrade env-injector-webhook env-injector-webhook --install --namespace <your namespace>
Note: As the pods and service need to have:
- a secret containing a signed certificate and key
- a mutating webhook patched with the CA Bundle
the script executed from
pre-install-job.yamltakes care of creating them executing as a helm pre-install + pre-upgrade hook. This allows the installation/upgrade steps to execute in the right order, but has the (unfortunate) side effect of leaving around the secret and mutating webhook when the chart is deleted. For that reason a pre-upgrade + post-delete helm hook takes care of deleting secret and admission webhook.
If you wish to update or increase the coverage of this webhook you can use the following API Guide for Kubernetes and Golang:
This repo is based on the excellent tutorial available at: morvencao/kube-mutating-webhook-tutorial
![renovate[bot] avatar](https://avatars.githubusercontent.com/in/2740?v=4 "renovate[bot]")
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent