Now that our Mosquitto pod is deployed we can set up a Reloader pod. What this Reloader instance will do is monitor all the namespaces in our cluster, looking for annotations in other application manifests, namely configMap
and secret
manifests. These annotations let Reloader know that we want to track the deployments for changes in the configMaps
and secrets
and then redeploy these deployments
after a change is detected in order to keep our pods up to date automatically.
The first thing to do is ensure you're logged into your cluster, then create a new local directory to work on the reloader deployment. Also, let's make a new namespace called reloader in our cluster:
$ oc new-project reloader
To deploy Reloader we will use the kustomize install instructions on the reloader github page: relaoder
So we will just copy this code into a kustomization.yaml file for now:
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
bases:
- https://github.com/stakater/Reloader/deployments/kubernetes
namespace: reloader
Let's see if we can deploy our reloader pod
with our kustomize file as is. To do that make sure you're in the directory with your kustomization.yaml and run:
$ kubectl apply -k .
serviceaccount/reloader-reloader created
clusterrole.rbac.authorization.k8s.io/reloader-reloader-role created
clusterrolebinding.rbac.authorization.k8s.io/reloader-reloader-role-binding created
deployment.apps/reloader-reloader created
Okay that's a good start, we didn't get any errors on the command line. Let's double check that everything is up and running:
$ oc get deployments NAME READY UP-TO-DATE AVAILABLE AGE reloader-reloader 0/1 0 0 6m44s
Okay so something is preventing our deployment from going up. When debugging sometimes I find it nice and easy to debug at the web UI but there are also highly detailed ways to debug using oc or kubectl at the command line. Time to take a closer look at the deployment using the web UI.
One of the first places you can look for hints when something goes wrong with your deployment is the details tab of said deployment in the conditions section. Here we can see a security context issue is causing our deployment to fail:
![seccontext](https://user-images.githubusercontent.com/84931428/148431566-b3a8e296-25b9-401f-8a90-c99fdbe5af1e.png)
Specifically, the message tells us we need to look at spec.containers[0].securityContext in our deployment.yaml:
![seccontextyaml](https://user-images.githubusercontent.com/84931428/148432933-3eac1233-8d60-4a93-88df-cb2bb4f37b05.png)
The reason we're getting this error is because the reloader deployment is set up with a user that doesn't exist in our environment. To get around this we can simply edit the security context (under spec.template.spec in the deployment yaml manifest) from this:
...
securityContext:
runAsNonRoot: true
runAsUser: 65534
...
To this:
...
securityContext: {}
...
You can edit the manifest right in the YAML tab of your reloader-reloader deployment
to see what affect this will have. After you make the change make sure to hit the save button so the deployment refreshes. Now our pod shows up!... Click the Pods tab to take a look. But we are seeing another error now, this time an ImagePullBackOff error.
You'll probably see this a lot when testing in quicklab but the fix is relatively simple. If you want to see a more detailed description of this error click on your pod and look into the Events tab. This tab is very helpful for debugging purposes.
![imagepullbckoff](https://user-images.githubusercontent.com/84931428/148437133-e349b751-99f5-45c0-9807-aa2403eb4f93.png)
As we can see Docker is rate limiting us while attempting to pull the image so we need another way to pull the image. To get around this rate limiting issue we need to use Quay.io. Quay much like Docker allows us to push and pull images, just without the pesky Docker rate limiting.
You will need to either create or login to your quay account at: quay.io
Now let's create a new repository in our quay account which we will use to store our image. Click the "Create New Repository" button and choose the following configuration:
![quay](https://user-images.githubusercontent.com/84931428/148837954-db320d26-dc65-489e-8fdb-05865080945b.png)
Once you're all setup over there let's look at the Docker Hub website to see the image we were trying to pull: DockerHub Reloader. Be sure to note the version number (v0.0.104 in case) in the error message of you pod.
Now login to quay at our terminal using podman and pull the image you need to your local machine
$ podman login quay.io #use your quay username and pwd
$ podman pull stakater/reloader:v0.0.104 #pull the image using the appropriate version number
$ podman images #Run this to get the image ID of the image you just pulled
$ podman tag <YOUR_IMAGEID> quay.io/QUAY_USERNAME/reloader:v0.0.104 #tag the image so we can push it to quay
$ podman push quay.io/QUAY_USERNAME/reloader:v0.0.104 #Now push the container image to your quay reloader repo
With our quay repo now set up we can point our deployment to our it so we can actually pull the Reloader image. To do so we need to edit the spec.containers.image field from this:
...
imagePullPolicy: IfNotPresent
terminationMessagePolicy: File
image: 'stakater/reloader:v0.0.104'
...
To this:
...
imagePullPolicy: IfNotPresent
terminationMessagePolicy: File
image: 'quay.io/USER_NAME/reloader:v0.0.104'
...
Once you make the change to the deployment yaml file click save again and view your pod. It's status now says running! So the changes we've made to the container image source and the security context have gotten our reloader pod to a running state. Let's now use Kustomize to declaratively make these changes, so we don't need to manually configure the deployment in our cluster every time we deploy it. To do this we will make a patch of our kustomization.yaml. This in essence uses the original kustomization file from stakater and overwrites it with changes we specify in our patch.
Lets create a directory within our working directory called patches. Within that folder let's create a document called deployment_patch.yaml. It should look like this:
deployment_patch.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: reloader-reloader
spec:
template:
spec:
securityContext: NULL
containers:
- image: quay.io/dystewar/reloader:v0.0.104
name: reloader-reloader
Switching back to our main directory we need to update our kustomization.yaml:
kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
bases:
- https://github.com/stakater/Reloader/deployments/kubernetes
patchesStrategicMerge:
- patches/deployment_patch.yaml
namespace: reloader
Now lets delete our deployment reloader-reloader and redeploy with our updated kustomization.yaml that reflects our patches:
$ oc delete deployment reloader-reloader
$ kubectl apply -k .
Now our reloader deployment is successfully configured, without having to make the manual changes we made before. Our patch took care of that for us. Great so everything is up and running, but now what? Let's check the logs tab of our reloader pod
to make sure everything is looking good within the pod itself:
![logs](https://user-images.githubusercontent.com/84931428/148869201-19e5c1b7-dc82-4f19-9f41-5e46ca8a1167.png)
Okay what on Earth is this trying to tell us? Well it mentions a serviceaccount
, which is used by processes in our pods to provide a flexible way to control API access without sharing a regular user’s credentials. Further analysis shows that our serviceaccount
can't access deploymentconfig
resources. So there's something going on here with our role based access control (rbac) here so we'll need to make a change to our clusterrole
. Before we do that make sure to take a look at this link and look specifically at the sections regarding clusterroles
and clusterrolebindings
: rbac
If we take a look in the Reloader repo at the clusterrole
: clusterrole.yaml, specifically at the rules section, we don't see deploymentconfig rule. Permissions using rbac are additive, meaning permissions are denied by default, so now it makes sense why we were getting that error in the pod
logs. Also note the name of the clusterrole
(reloader-reloader-role).
You might be wondering how we know that our serviceaccount
is associated with this clusterrole
. The easiest way to see this is by running:
oc get sa
NAME SECRETS AGE
builder 2 5d3h
default 2 5d3h
deployer 2 5d3h
reloader-reloader 2 5d3h
So the serviceaccount
we have been talking about is called reloader-reloader.
Now take a look at clusterrolebinding.
The roleRef section shows we are referencing our (reloader-reloader-role) clusterrole
and the subject section is binding that clusterrole
to our (reloader-reloader) serviceaccount
.
Now we just need to add permissions for accessing deploymentconfigs
to our clusterrole
. We can do this creating another Kustomize patch to our deployment:
clusterrolepatch.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: reloader-reloader-role
rules:
- apiGroups:
- "apps.openshift.io"
resources:
- deploymentconfigs
verbs:
- list
- get
- update
- patch
Add the above file to your patches folder.
Then update your kustomization.yaml:
kustomization.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: reloader-reloader-role
rules:
- apiGroups:
- ""
resources:
- secrets
- configmaps
verbs:
- list
- get
- watch
- apiGroups:
- "apps"
resources:
- deployments
- daemonsets
- statefulsets
verbs:
- list
- get
- update
- patch
- apiGroups:
- "extensions"
resources:
- deployments
- daemonsets
verbs:
- list
- get
- update
- patch
- apiGroups:
- "apps.openshift.io"
resources:
- deploymentconfigs
verbs:
- list
- get
- update
- patch
Finally lets redeploy to fix our pod:
$ oc delete deployment reloader-reloader
$ kubectl apply -k .
Now we can see the logs of our Reloader pod
and everything appears normal! We can see that Reloader is now monitoring all our namespaces in the cluster for changes in configmaps
and secrets
.
![monitoring](https://user-images.githubusercontent.com/84931428/149029956-e1691850-c93b-45dd-80e7-b44fb1faf0ea.png)