- az cli (latest version: 2.24.x +): https://docs.microsoft.com/en-us/cli/azure/install-azure-cli
Info: Issues when running
az aks create
when the az cli version is not updated. If you experience an issue similar to the following, try updating the az cli.
- kubectl (version 1.19.x +): https://kubernetes.io/docs/tasks/tools/
- docker (version 20.10.x +): https://docs.docker.com/engine/install/
Reference: https://docs.microsoft.com/en-us/azure/aks/concepts-clusters-workloads
-
Control Plane
The Control Plane deployed for your Kubernetes cluster is a managed Azure resource that is abstracted away from the user. You will not configure the control plane and cannot access it directly (access is done through kubectl or the Kubernetes Dashboard).
-
Node Pools
Nodes of the same type and configuration are grouped together into node pools on AKS. You may have a default set of nodes and then can add a unique/specific node pool to include different VM types into your cluster for specialized workloads.
-
Taints and Tolerations
Taints will restrict the types of pods that can be scheduled on the nodes tagged with the specified taints. Apply tolerations to a pod so that the pod can then be scheduled on nodes with that taint.
-
Cluster Autoscaler
The cluster autoscaler can dynamically add and remove nodes from specific node pools to align to your resource utilization and scheduling demands.
Reference: https://docs.microsoft.com/en-us/azure/aks/cluster-autoscaler#using-the-autoscaler-profile
-
Container Storage Interface (CSI)
This is an interface or standard by which 3rd-party storage solutions can plug into Kubernetes. For AKS, this currently needs to be enabled at cluster creation time until the feature is GA.
Reference: https://docs.microsoft.com/en-us/azure/aks/csi-storage-drivers
-
Save the following environment variables in a
.env
file:# Azure resoure group settings RG_NAME=demo-aks-cluster # Resource group name LOCATION=eastus # az account list-locations --query '[].name' # ACR deployment settings ACR_NAME=demoacrinstancehs123 # ACR instance name ACR_SKU=basic # ACR sku # AKS deployment settings AKS_NAME=demo-aks # AKS Cluster Name, ie: 'demo-aks' AKS_NODE_COUNT=2 # Initial Node Count in Cluster
-
Run the following:
# Login to your Azure Subscription az login # Source and export the environment variables set -a source .env # Assumes your .env is at ./src/drupal-aks/.env set +a # Register the EnableAzureDiskFileCSIDriver preview feature before deploying AKS az feature register --namespace "Microsoft.ContainerService" --name "EnableAzureDiskFileCSIDriver" # Verify the registration status (this takes a few minutes before you should see Registered) az feature list -o table --query "[?contains(name, 'Microsoft.ContainerService/EnableAzureDiskFileCSIDriver')].{Name:name,State:properties.state}" # Refresh the registration of the Microsoft.ContainerService resource az provider register --namespace Microsoft.ContainerService # Install the aks-preview extension az extension add --name aks-preview # Update the extension to make sure you have the latest version installed az extension update --name aks-preview # Deploy the initial cluster ./deploy.sh # Get kubectl credentials az aks get-credentials --name $AKS_NAME --resource-group $RG_NAME --overwrite-existing
-
View the system nodepool:
az aks nodepool list --cluster-name $AKS_NAME -g $RG_NAME
-
Deploy another node pool:
Info: You cannot change the taint for the node pool after the pool is created.
az aks nodepool add \ --resource-group $RG_NAME \ --cluster-name $AKS_NAME \ --name cpupool \ --node-count 1 \ --labels hardware=highcpu \ --node-taints sku=gpu:NoSchedule
-
Run
kubectl get nodes
to view the new node. Also runaz aks nodepool list --cluster-name $AKS_NAME --resource-group $RG_NAME -o table
to view the node pool.Info: The node pools can also be seen in the Portal for the cluster
-
Run the following:
kubectl apply -f app-workload.yaml kubectl get events --watch
-
View the Kubernetes Portal in Azure and see that the pods are scheduled on the new node pool given the taints and tolerations that were applied.
-
Run the following:
az aks nodepool scale \ --resource-group $RG_NAME \ --cluster-name $AKS_NAME \ --name cpupool \ --node-count 2
-
Confirm in the portal that the node pool has been scaled up by a node.
-
Scale the node back down in the portal and confirm the action executes successfully with
kubectl get nodes
. -
Confirm the app is still up and running.
-
Configure cluster autoscaling for the node pool by running the following command (before running the command, confirm there is only one node in the cpupool node pool):
az aks nodepool update \ --resource-group $RG_NAME \ --cluster-name $AKS_NAME \ --name cpupool \ --enable-cluster-autoscaler \ --min-count 1 \ --max-count 5
-
Run
kubectl delete deployment/azure-vote-front
andkubectl delete deployment/azure-vote-back
. -
Run
kubectl apply -f app-workload-replicas.yaml
. Confirm after running this that additional nodes are added to the cpupool. -
Additionally run
kubectl scale deployments/azure-vote-front --replicas=10
and you will see additional nodes added by the cluster autoscaler. -
Remove the deployments by running the following:
kubectl delete deployments/azure-vote-front kubectl delete deployments/azure-vote-back
Info: In about 10 minutes, per the default cluster autoscaling profile referenced below, nodes should begin to be removed from the node pool.
Reference: https://docs.microsoft.com/en-us/azure/aks/cluster-autoscaler#using-the-autoscaler-profile
-
When you deployed the AKS cluster, we included the CSI drivers for Azure-based storage (Azure Files and Azure Disks).
-
Run
kubectl get sc
to view the storage classes available in the cluster. We will run a test with the azurefile-csi storage class.Reference: https://docs.microsoft.com/en-us/azure/aks/azure-files-csi
-
Run the following to create the PVC and an nginx pod that uses that PVC:
kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/azurefile-csi-driver/master/deploy/example/pvc-azurefile-csi.yaml kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/azurefile-csi-driver/master/deploy/example/nginx-pod-azurefile.yaml
-
View the volume in the portal and notice that the volume is mounted at
/mnt/azurefile
. -
Run
kubectl exec --stdin --tty nginx-azurefile -- /bin/bash
to open a shell in the running container. From there, navigate to/mnt/azurefile
and create a test file withtouch testfile.txt
. -
In the Azure portal, go to the
MC_
resource group and open up the azure file shares tab under the storage account. You should see the test file in the file share.
-
Run
az acr login --name $ACR_NAME
to login to the registry. -
Run the following to pull the images to deploy locally:
docker pull mcr.microsoft.com/oss/bitnami/redis:6.0.8 docker pull mcr.microsoft.com/azuredocs/azure-vote-front:v1
Info: These are the images from the test app previously used.
-
Run the following to tag the images so they can be pushed to the ACR instance:
# Get the registry server az acr list -g $RG_NAME -o table # Tag the images docker tag mcr.microsoft.com/oss/bitnami/redis:6.0.8 $ACR_NAME.azurecr.io/demo/backend docker tag mcr.microsoft.com/azuredocs/azure-vote-front:v1 $ACR_NAME.azurecr.io/demo/frontend
-
Push the images to ACR:
docker push $ACR_NAME.azurecr.io/demo/backend docker push $ACR_NAME.azurecr.io/demo/frontend
-
Run
az acr repository list --name $ACR_NAME
to view the repositories. -
Open the
app-workload-acr.yaml
file and update the images to reference your ACR instance. -
Run the following to test the deployment in a new namespace:
# Create the namespace kubectl create ns acr-demo # Deploy the service and deployments kubectl apply -f app-workload-acr.yaml -n acr-demo # Run a port-forward to view the front-end from your local machine (usefule when not deploying as a LoadBalancer service) kubectl port-forward svc/azure-vote-front 8000:80 -n acr-demo
Info: Assumes the Azure File CSI Driver is installed from lab 1
Reference: https://github.com/kubernetes-sigs/azurefile-csi-driver/tree/master/deploy/example/nfs
-
Run the following to Allow NFS File Shares under your subscription:
az feature register --name AllowNfsFileShares --namespace Microsoft.Storage # Run the command below to validate the feature is registered # This may take up to 30 minutes # You will see 'Microsoft.Storage/AllowNfsFileShares Registered' once it is complete az feature list -o table --query "[?contains(name, 'Microsoft.Storage/AllowNfsFileShares')].{Name:name,State:properties.state}" az provider register --namespace Microsoft.Storage
-
Run the following to create a storage class that will dynamically provision an NFS File Share when called through a PVC:
wget https://raw.githubusercontent.com/kubernetes-sigs/azurefile-csi-driver/master/deploy/example/storageclass-azurefile-nfs.yaml kubectl create -f storageclass-azurefile-nfs.yaml
-
Run
kubectl create ns nfs-test
-
Run
helm install wordpress -f wordpress-values.yaml -n nfs-test bitnami/wordpress
-
Run
kubectl logs -f $(kubectl get pod -l app.kubernetes.io/instance=wordpress -o jsonpath='{.items[0].metadata.name}' -n nfs-test) -n nfs-test
to follow the logs of the container as it starts (takes about 10 minutes on a fresh install based on my testing).
-
Register the nfs v3 feature
az login # regiser the nfsv3 feature in the subscription az feature register --namespace Microsoft.Storage --name AllowNFSV3 # verify that the feature is registered az feature show --namespace Microsoft.Storage --name AllowNFSV3 # once registered, run this command to register the resource provider az provider register -n Microsoft.Storage
-
Create a Storage Account Properly Configured for NFS
Info: Before creating the storage account, you must have registered the feature as shown above.
Run through step 5 of this tutorial in the Azure Portal - be sure to properly configure the Storage Account so that the NFS setting can be enabled.
Info: Notice the hierarchical namespace setting as well as the networking setting. You can either use a private endpoint or just select public endpoints (selected networks) and accept the VNet where AKS is deployed. This will secure the storage account to only allow connections from the AKS VNet.
-
Create a container in the storage account for the blob
Follow step 6 - there is no special rule for how the container is created to support the backing blob storage.
-
In your cluster, run through the following steps:
Info: Review the comments before running the commands to properly update the templates to work for your NFS storage account
kubectl create ns nfs-blob-sample # Open nfs-blob-templates/nfs-blob-pv.yaml and update the path and server settings to configure the NFS Volume kubectl apply -f nfs-blob-templates/nfs-blob-pv.yaml -n nfs-blob-sample kubectl apply -f nfs-blob-templates/nfs-blob-pvc.yaml -n nfs-blob-sample kubectl apply -f nfs-busybox-pod.yaml -n nfs-blob-sample # Once the busybox pod is up and running, exec into the pod and view the "/mnt" folder # Create a file in the folder kubectl exec -it pod/busybox-sleep sh -n nfs-blob-sample cd /mnt echo "hello world" > hello-world.txt ### # Run through the same steps for another busybox pod so we can see how multiple pods can talk to the same NFS backed blob ### # Open nfs-blob-templates/nfs-blob-pv-two.yaml and update the path and server settings to configure the NFS Volume kubectl apply -f nfs-blob-templates/nfs-blob-pv-two.yaml -n nfs-blob-sample kubectl apply -f nfs-blob-templates/nfs-blob-pvc-two.yaml -n nfs-blob-sample kubectl apply -f nfs-busybox-pod-two.yaml -n nfs-blob-sample # Once the busybox pod is up and running, exec into the pod and view the "/mnt" folder # View the previously created file kubectl exec -it pod/busybox-sleep sh -n nfs-blob-sample cd /mnt cat hello-world.txt
There are a few options when it comes to ingress with AKS. We will look at the nginx ingress controller and application gateway ingress controller.
-
Run
kubectl create namespace ingress-basic
-
Add the helm repo for ingress-nginx:
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
-
Run the following helm install to deploy the ingress controller:
Info: Here is the repo to further configure the ingress controller deployment.
Info: On the Azure side, you will find a Public IP is provisioned on the Load Balancer associated with your cluster. This will be the front-end IP that fronts the nginx ingress controller.
helm install nginx-ingress ingress-nginx/ingress-nginx \ --namespace ingress-basic \ --set controller.replicaCount=2 \ --set controller.nodeSelector."beta\.kubernetes\.io/os"=linux \ --set defaultBackend.nodeSelector."beta\.kubernetes\.io/os"=linux \ --set controller.admissionWebhooks.patch.nodeSelector."beta\.kubernetes\.io/os"=linux
-
To test the ingress controller, run the following:
# Deploy the resources
kubectl create ns test-nginx-app
kubectl apply -f aks-helloworld.yaml -n test-nginx-app
# Test the ingress
# Navigate to the address shown here
kubectl get ingress -n test-nginx-app
In this lab, we will leverage the AKS add-on feature to deploy our app gateway ingress controller. Another deployment method is through helm.
Here are some of the Benefits of App Gateway Ingress Controller.
With AGIC you can deploy it as an add-on at cluster creation time or add it to an existing AKS cluster. Additionally, you can either have the add-on provision an App Gateway instance or use an existing App Gateway instance.
- Run
az aks enable-addons --name <AKS-NAME> --resource-group <RESOURCE-GROUP> -a ingress-appgw --appgw-name myApplicationGateway --appgw-subnet-cidr "10.2.0.0/24"
Info: App Gateway requires a dedicated subnet. The command above will provision that subnet within the same default VNet used with AKS, but you may need to modify this depending on your AKS deployment. This reference further describes the infrastructure/networking configuration for App Gateway.
Validation: If you look into the
MC_
resource group that holds the infrastructure associated with the AKS cluster, you should now see an App Gateway resource provisioned.
Validation: If you view the App Gateway resource provisioned for the cluster, you should see an 'Updating' status. The App Gateway will take some time to get configured and setup for use.
- Run the following:
# Deploy the resources
# If you review the aks-helloworld-appgateway.yaml file, you'll see that changes were made on annotations to specify app gateway:
# kubernetes.io/ingress.class: azure/application-gateway
kubectl create ns test-appgateway-app
kubectl apply -f aks-helloworld-appgateway.yaml -n test-appgateway-app
# Test the ingress
# Navigate to the address shown here
kubectl get ingress -n test-appgateway-app
-
Install CSI Driver with Helm
Important: Be sure to follow the prerequisites here for this to work
helm repo add blob-csi-driver https://raw.githubusercontent.com/kubernetes-sigs/blob-csi-driver/master/charts helm install blob-csi-driver blob-csi-driver/blob-csi-driver --namespace kube-system
-
Setup the following storage class
You can also provision a storage account manually as described here
kubectl apply -f blob-storage-class.yaml
-
Create an application that uses the storage class. Here is an example statefulset that mounts a volume:
kubectl create -f https://raw.githubusercontent.com/kubernetes-sigs/blob-csi-driver/master/deploy/example/statefulset.yaml
You can validate it worked by going into the Azure Portal and viewing the storage account. You should see a file being updated regularly.
To see the mount in the contianer
kubectl exec -it statefulset-blob-0 -- df -h