octopusknives / home-cluster Goto Github PK

My home Kubernetes Talos cluster ⛵

... managed with Flux 🤖

This repository is my home Kubernetes cluster in a declarative state. Flux watches my cluster directory and makes the changes to my cluster based on the YAML manifests.

Feel free to open a Github issue if you have any questions.

This repository is built off the k8s-at-home/template-cluster-k3s repository.

• Talos: Built using talhelper
• fluxcd: Sync kubernetes cluster with this repository.
• SOPS: Encrypts secrets which is safe to store - even to a public repository.

• Cilium: For internal cluster networking, also as load balancer to expose services.
• cert-manager: Configured to create TLS certs for all ingress services automatically using LetsEncrypt.
• traefik: Ingress controller for services.
• authelia: Full featured authentication server.

• rook-ceph: Cloud native distributed block storage for Kubernetes
• nfs-subdir-external-provisioner: Provides persistent volumes from NFS server.

• Intel GPU plugin: Access intel gpu available on nodes.
• node-feature-discovery: Discover features available on nodes.
• smarter-device-manager: Access devices available on nodes.

• Prometheus: Scraping metrics from the entire cluster
• Grafana: Visualization for the metrics from Prometheus

The Git repository contains the following directories under cluster and are ordered below by how Flux will apply them.

./cluster
├── ./base    # entrypoint to Flux
├── ./config  # cluster config, loaded before `core`
├── ./crds    # custom resource definitions (CRDs), loaded before `core`
├── ./core    # important infrastructure applications, loaded before `apps`
└── ./apps    # common applications, loaded last

Incoming http and https traffics from outside of my network are forwarded from OPNSense firewall into Traefik pod with a LoadBalancer service using MetalLB layer2 implementation. So, basically this is how the http(s) traffic flows:

Internet -> OPNSense firewall -> Traefik service -> Kubernetes pod

Traefik service is using Local externalTrafficPolicy so I can track the real IP of clients trying to access my services. For important backend services like my OPNSense and Traefik dashboards, I use ipWhiteList middleware to only allow access from my internal networks. My certificates are managed with cert-manager using LetsEncrypt as the CA.

Secrets are encrypted using sops before being pushed into this repository. The encrypted secrets are then decrypted by sops using the private key inside the cluster. For encryption/decryption, I use age. The public key to encrypt the secret is in .sops.yaml. Secrets environment variables for the cluster are in cluster-secret-vars.yaml. The non secret variables are in cluster-vars.yaml.

Metrics scraping for the cluster are done using Prometheus.

Dashboards included in my cluster are:

• The provided dashboard from kube-prometheus-stack
• Traefik dashboard from grafanalabs
• Fluxcd dashboard from here
• Rook-ceph dashboards from here

To add your own dashboard, create a configmap with the data include the json file of the dashboard and add label grafana_dashboard: "1" to the manifest. The sidecar container from this image will mount the dashboard into your grafana pod.

A lot of inspiration for my cluster came from the people that have shared their clusters over at awesome-home-kubernetes

• Use redis operator