This script is not for Prod rather a 1-Click way to setup Kubernetes cluster along with its various components in homelab (or other non critical environments). Although it does support multiple stacked control planes and follows certain HA principles specified in Kubernetes documentation, I have not tested it extensively. Use it at your own risk.

I created this for my homelab and sharing in hope that it would enable others to set things up quicker. The script currently supports CentOS7 and CentOS 8 and I tested these 2 in my homelab. I was able to mix and match nodes without any issue.

While trying to setup my first cluster, I ran into lot of issues and hopefully this script would save you from all of those. I added resolutions to script with documentation to avoid future pain :) .

Please set up public key based SSH access to all nodes (provisioned VMs) you are planning to use and connect to those nodes atleast ones via SSH. This would minimize the number of prompts (only 2 prompts) you would get during script execution. Get block of available IP address from your local router. In case of DD-WRT based router, connect to the router via SSH and run cat /var/lib/misc/dnsmasq.leases to get a list of IP in use. As always I would recommend key based SSH for router as well. Setup local DNS provider (in my case I use Pihole) to provide a local URL that would be used by various dashboard exposed by our MetalLB + Ingress controller combo. Also make sure that raw/block drive is attached to worker nodes for Ceph to format and use.

If something is not clear, I would recommend to download the script and take a look as I have added comments to make it easy for others to understand and customize.

The script is relatively self contained and fetches necessary files from github repos. To execute the script successfully we need:

To keep it as streamlined as possible, I have defined variables in main script that can be updated to align with local environment. All other scripts (setting up storage, setting up monitoring etc.) are sourced in main script. Thus we only need to update variables in the main script (Setup_Kubernetes_V01.sh)

Before setting up Kubernetes nodes, we set up KeepAlived and HAProxy to provide virtual ip address for control plane (used by Kubernetes cluster later). This script installs and configures all prerequisites needed by KeepAlived and HAProxy. The script supports 3 modes:

Main script calls prepare_node.sh to set up all prerequisites of nodes in a Kubernetes cluster. DNS/IP based access to nodes, SELinux setting, Swap disable, Firewall, IP Forwarding, kernel mods, CRI-O/Containerd, Docker, Kuberadm, Kubelet, Kubectl and anything else required by a node in a Kubernetes cluster is setup by this script for all nodes (Master as well as workers).

Once nodes are setup, script initializes the primary node ( with flags: --control-plane-endpoint, --pod-network-cidr and --upload-certs) sets up networking (Calico and Weave supported via flags), adds other master nodes (defined in MASTER_NODE_NAMES) and adds worker nodes to the cluster (defined in WORKER_NODE_NAMES). Note: kubectl get cs would show error due to a known Kubernetes bug.

Once the cluster is ready, the script deploys MetalLB to act as load balancer (External IP would populate for services utilizing LoadBalancer). The script waits till MetalLB is ready before proceeding. MetalLB allocates IP addresses within IP range defined by START_IP_ADDRESS_RANGE and END_IP_ADDRESS_RANGE.

Next the script deploys Nginx as ingress controller. Using MetalLB, the ingress controller gets external ip address and exposes services/dashboards.

Based on flag (SETUP_ROOK_INSTALLED), script calls setup_rook_ceph.sh to:

Based on flag (SETUP_CLUSTER_MONITORING), script calls setup_monitoring.sh to:

The INGRESS_DOMAIN_NAME is used during config and all dashboards are accessible on the generated URL.

After completing the prerequisite step defined above, connect to your server (terminal or SSH session), got your home directory and download the script using: wget https://raw.githubusercontent.com/piyushkumarjiit/K8S/master/Setup_Kubernetes_V01.sh

Update the permissions on the downloaded file using: chmod 755 Setup_Kubernetes_V01.sh

Update below mentioned variables in the script using an editor:

Now run the script and follow prompts: sudo ./Setup_Kubernetes_V01.sh |& tee -a setup.log

If everything went well so far, we would have a working HA Kubernetes cluster with external load balancer, storage, ingress and monitoring. We would be able to login to Grafana dashboard (using admin/admin) from any computer in local network. We can login into other dashboards (Rook, Prometheus and AlertManager) as well.

We can add new nodes to the cluster without rerunning the script. To add a node, copy the prepare_node.sh to the node to eb added, update the variables in prepare_node.sh to be only applicable to the specific node and execute prepare_node.sh. Once it completes successfully, copy the applicable node on-boarding command from primary node (the node from where you ran the script). In the HOME folder there would be 2 files add_worker.txt and add_master.txt which contain respective commands. Execute these commands from your new node and it would be added to the cluster.

I have also created set of scripts that would cleanup and bring your nodes to original state (almost).

Connect to your server (terminal or SSH session), got your home directory and download the script using: wget https://raw.githubusercontent.com/piyushkumarjiit/K8S/master/Cleanup_Kubernetes_V01.sh

Update the permissions on the downloaded file using: chmod 755 Cleanup_Kubernetes_V01.sh

Update the variables(mentioned in Installation section above) updated for Setup_Kubernetes_V01.sh using an editor.

Now run below script and follow prompts: sudo ./Cleanup_Kubernetes_V01.sh |& tee -a cleanup.log

This is just a start and the script(s) could be further improved. NFS could be added to Ceph/storage via automated shell scripts. You can update the script to add firewall rules in place of disabling Firewalld. I am planning to add Ubuntu and Raspberry pi support down the line (if people find it useful).

Piyush Kumar - (https://github.com/piyushkumarjiit)

This project is licensed under the Apache License - see the LICENSE.md file for details

Thanks to below URLs for providing me the necessary understanding and material to come up with this script.