Smart Home on K8s

A Smart Home running on Kubernetes (for High Availability … and fun! 🎉 )

Table of Contents

Hardware
Operating system
- Raspbian, with passwordless SSH access
Preparations
Install K3s (lightweight Kubernetes)
Install Smart Home applications

Hardware

I am using:

Raspberry Pi 4 (4gb)
Raspberry Pi 4 (8gb)
Raspberry Pi 3 (1gb)
- (more raspberries coming, when I can finally find more pi to buy)
3 x SanDisk High Endurance 128GB SD cards

Operating system

Raspbian, with passwordless SSH access

Burn Raspbian Bullseye on an SD card, as per: https://www.raspberrypi.com/software/
Create an empty file called ssh on the root folder of the SD card. This enables SSH, which is crucial if you don’t have your raspberry pi connected to a screen. Default username: pi, default pass: raspberry.
Run sudo raspi-config to connect your PI to the Wi-Fi, if not connected by Ethernet. Also change the default password.
Enable passwordless ssh, by copying your public key to the raspberry pi’s ~/.ssh/authorized_keys file. For convenience, you can also use ssh-copy-id -i $PATH_TO_KEY pi@$PI_IP_ADDRESS
Ensure you have added the key to your agent with ssh-add -K $PATH_TO_KEY. Now try to connect via SSH and ensure you don’t get a password prompt.

Preparations

The first thing you’ll need for the cluster setup is to install Ansible on your workstation. Your workstation will connect to each of the raspberry pi nodes via SSH and run all the commands in the Ansible playbook, as you sip your beverage of choice relaxing on your chair.

You can install Ansible following the guide here: Ansible Installation Guide

You’ll also need git cli in order to checkout the project and run k3s-install (see below). In case it isn’t already installed (try git --version), something like sudo apt install git-all will do the job.

Finally, please have in mind that you need to change the hostname(s) of the pods where all the service will be installed. So before any make command just be sure that you’ve edited the appropriate file replacing my hostname(s) with yours.

Install K3s (lightweight Kubernetes)

I was looking for an Ansible playbook that supports an HA setup of K3s, so that the applications we deploy can actually tolerate hardware failures of the individual Raspberry Pi nodes.

The biggest choice regarding the HA setup is whether you will use an external database, or you will go with the newer option of an embedded etcd cluster.

Considering I didn’t have any other HA database at home, I went for the simplest - in terms of initial setup - option: the embedded etcd cluster.

The k3s-install Makefile target will point you to the ansible playbook used.

Before running make k3s-install you’ll need to install ansible role for k3s. More info here

You’ll also need to alter k3s/ansible/inventory.yaml so it depict your hardware inventory.

Access your shiny new cluster

After the playbook is finished and everything has gone well, you need to access the cluster.

You can ssh into one of your PIs and copy the contents of /etc/rancher/k3s/k3s.yaml.

Paste the contents of this file into e.g. ~/.kube/k8spi and use that as your KUBECONFIG file. Make sure to edit 127.0.0.1 or localhost in that file and replace it with the IP of your control_plane node.

export KUBECONFIG=~/.kube/k8spi

Then, you should see:

Once you have everything ready, it’s time to start deploying stuff on our Kubernetes K3s cluster!!

But we need a bit more setup first: a Dashboard, to have a web-based UI, and the NFS provisioner, so that we can have some persistent storage.

Setup Persistent Storage

For persistent storage, I have:

Enable NFS on the Synology NAS: Control Panel → File Services → NFS
Created a Shared Folder (called … "Kubernetes"). All PersistentVolumes will be created in this folder.
Allowed access to this shared folder to be accessed by the cluster IPs (see screenshot below)
Used the NFS subdir provisioner, which I have included in this repo.
Edit nfs-subdir/standard/deployment.yaml and add your own NFS_SERVER and NFS_PATH values
Install with make nfs-install.

Please note that there is also another direcotry called fast inside nfs-subdir. You can safely ignore it as it is used only if you need a second NFS storage let’s say based on faster SSD disks. If you need it you can uncomment the respective line in the Makefile and make the changes in the deployment.yaml file.

Kubernetes dashboard

make dashboard

To log in to the dashboard, you need to create a token:

kubectl --namespace kubernetes-dashboard create token admin-user

Use the token to log in, or add an entry to your KUBECONFIG:

    token: <the value you got from the above command>

Install Smart Home applications

With all that out of the way…

It’s finally(!!) time to start making our home smarter!!! 🎉 🥳 🚀

BUT BEFORE YOU START!!!

DNS Config

In my manifests, you will find that I am using *.k3s.yorgos.net.gr domains, to expose services running within the K3s cluster.

In order for that to work, you need to add A records for your corresponding wildcard domain. I have 3 k3s cluster nodes, so I added 3 A records - one for each local IP address.

Pi-hole

Pi-hole is a network-based ad-blocking piece of software. It is also a custom DNS server (more on this in a bit).

I deploy pi-hole because I can’t run uBlock Origin on every browser of every device (not all of them are controlled by me 😅), so having a way to block ads on the network level is 🎉!

It uses DNS sinkholing and blocklists as a way of stopping internet ads, malware, malvertising, etc.

Apart from the ad-blocking functionality, running a local DNS server helps me with one more thing: DNS resolution on Internet-blocked devices.

As you may have noticed, I use public DNS entries to map to local IP addresses: e.g. all my apps are deployed under *.k3s.yorgos.net.gr.

You will need to create a secret in the pihole namespace (create it if it doesn’t exist):

---
apiVersion: v1
kind: Secret
metadata:
  name: pihole-secret
type: Opaque
stringData:
  password: some-super-secure-pass-for-your-pihole-web-ui

Once the secret has been created, you can:

make pihole

Mosquitto

I need my smart home automation software to access my smart devices (shelly/sonoff plugs, switches, etc. etc.) - for example, all my switches need to connect to the MQTT server (mosquitto). BUT - I block internet access on all these devices on my router! So, without a local DNS server, I would need to use a single IP address for my MQTT server and somehow ensure that a load balancer runs in High-Availability behind that IP address.

Instead of doing that, I can add 2 A records for mosquitto.k3s.yorgos.net.gr, for IP addresses 192.168.100.180 and 192.168.100.181 and my smart home setup will survive outages of one of the two nodes !! (I only run a single instance of mosquitto, but kubernetes will ensure it always runs on one of these two nodes and this way the clients will always find and connect to it!)

To install mosquitto itself, use:

make mosquitto

Home-Assistant

make home-assistant

After Home Assistant is installed, you will need to add the following section to your configuration.yaml (which probably lives in your NAS persistent volume folder):

# Uncomment this if you are using SSL/TLS, running in Docker container, etc.
# http:
#   base_url: example.duckdns.org:8123
http:
  server_host: 0.0.0.0
# optional ip_ban_enabled: true
# optional login_attempts_threshold: 5
  use_x_forwarded_for: true
  trusted_proxies:
  # Pod CIDR
  - 10.42.0.0/16
  # Node CIDR
  - 192.168.100.0/24

Nextcloud

For nextcloud, you will need to create 2 kubernetes secrets:

---
apiVersion: v1
kind: Secret
metadata:
  name: nextcloud-postgres-secrets
  labels:
    app: nextcloud-postgres
type: Opaque
stringData:
  POSTGRES_PASSWORD: "your postgres password"
  nextcloud-db-user: nextcloud
  nextcloud-db-password: "your nextcloud user db password"
---
apiVersion: v1
kind: Secret
metadata:
  name: nextcloud-secrets
type: Opaque
stringData:
  nextcloud-db-user: nextcloud
  nextcloud-db-password: "your nextcloud user db password"
  nextcloud-user: admin
  nextcloud-password: "some super secure pass for your admin user"

After you have created these 2 secrets (e.g. with kubectl apply ), in the nextcloud namespace, you can go ahead and run:

make nextcloud

Photoprism

Photoprism has become my "Google Photos" / "Apple Photos" privacy-friendly alternative solution.

First, you will need to create 2 kubernetes secrets. One for photoprism and one for mariadb - the backing database:

---
apiVersion: v1
kind: Secret
metadata:
  name: photoprism-mariadb-secrets
  namespace: databases
stringData:
  MARIADB_ROOT_PASSWORD: "some super clever root password"
  MARIADB_PASSWORD: "yet another password you shouldn't upload to github"
---
apiVersion: v1
kind: Secret
metadata:
  name: photoprism-secrets
  namespace: photoprism
stringData:
  PHOTOPRISM_DATABASE_DRIVER: "mysql"            # use MariaDB 10.5+ or MySQL 8+ instead of SQLite for improved performance
  PHOTOPRISM_DATABASE_SERVER: "photoprism-db:3306"     # MariaDB or MySQL database server (hostname:port)
  PHOTOPRISM_DATABASE_NAME: "photoprism"         # MariaDB or MySQL database schema name
  PHOTOPRISM_DATABASE_USER: "photoprism"         # MariaDB or MySQL database user name
  PHOTOPRISM_DATABASE_PASSWORD: "this should match with MARIADB_PASSWORD"       # MariaDB or MySQL database user password
  PHOTOPRISM_ADMIN_PASSWORD: "bla bla bla"

After you have created these 2 secrets (e.g. with kubectl apply ), you can go ahead and run:

make mariadb photoprism

Unifi Controller

After first buying the Unifi equipment, I ran the Unifi Controller on my laptop. But that wasn’t convenient because there were times I wanted to check my network configuration when I wasn’t at my laptop.

Then I moved it to my NAS, running on Docker Compose. But that wasn’t great because the Unifi controller is quite demanding in terms of resources and (I think!!) it caused my NAS to hang a couple of times < --- not good!!

So, finally, I decided to migrate it to the k3s cluster, for greater reliability.

To deploy it, just:

make unifi-controller

Shairport-Sync (Airplay)

Shairport-Sync is a great Airplay 1 emulator. I use it reliably (for a number of years), to turn an old set of roof speakers in my living room, into a "smart speaker", that I can stream music to over WiFi (not bluetooth).

In terms of deploying shairport-sync, the thing to be aware of is that we clearly only want it to run on a single raspberry of our raspberry pi cluster - the one that is actually connected to the speakers!

We can achieve that with the following combination:

Add a label to the node:

$ kubectl label nodes node3-k3s app=audio
node/node3-k3s labeled

Instruct the app to be deployed on the node with that label:

nodeSelector:
  app: audio

Mycroft Voice Assistant

Mycroft-specific instructions

Mimic3 Text-to-Speech engine

i.e. don’t let your mycroft sound like a robot !

Deploy mimic3 on k3s:

make mimic3

Once, Mimic3 is up and running, it is now time to point mycroft to it:

Use the following in your /home/pi/.config/mycroft.conf (you might need to move it to this path from /home/pi/.mycroft/mycroft.conf):

  "tts": {
    "marytts":{
       "url": "https://mimic3.k3s.yorgos.net.gr/",
#       "voice":"en_UK/apope_low"
# switched to the below, after my daughter's request
       "voice":"en_US/hifi-tts_low#92"
     },
    "module":"marytts"
  }

reboot

Traefik improvements

Run make traefik to:

expose Traefik dashboard for help with debugging
switches from Deployment to DaemonSet, to ensure traefik runs on all Raspberries
provides option to enable access log on traefik.

IMPORTANT: the traefik dashboard will be available at traefik.<your domain>/dashboard/. Do NOT forget the /dashboard/, including the final

What about Storage downtime?

There are times when your Network-Attached Storage (NAS) storage will go down.

Whether because you want to upgrade your NAS software, or because you want to change hard disks (and that requires downtime), or because of a power failure that exceeds your Uninterruptible Power Supply (UPS) limits.

In those cases, it is very useful to have a way to gracefully terminate any internal services that rely on Kubernetes persistent volumes provided by your NAS.

In this repo, you can use the nas-shutdown and nas-restart make targets.

HTTPS / TLS (with cert-manager and LetsEncrypt)

Choosing LetsEncrypt for non-self-signed (and non-paid-for) certificates comes with one caveat: we cannot use the http01 solver, because we don’t have any public endpoints that letsencrypt can verify the text on. We can, however, use the dns01 solver - with an additional dependency on Cloudflare (where my yorgos.net.gr domain is set up). cert-manager will use a secret API key to contact Cloudflare, and instruct it to set up the domains accordingly, in order to prove ownership to LetsEncrypt and allow the certificates to be issued correctly.

With that in mind, all the resources required are in the cert-manager/ folder.

We use:

the helm chart to deploy cert-manager
a letsencrypt staging ClusterIssuer, for our tests. LetsEncrypt applies rate limiting and it’s easy to go over during initial tests. Use letsencrypt-staging for all your tests and then switch your ingress to letsencrypt-prod once everything is fully working.
a traefik middleware to always redirect to HTTPS
a k8s secret to hold the Cloudflare "Global API Key" - do NOT use "API tokens" or the "Origin CA Key", like the one below:

---
apiVersion: v1
kind: Secret
metadata:
  name: cloudflare-api-key-secret
type: Opaque
stringData:
#  Cloudflare "Global API Key" - do NOT use "API tokens" or the "Origin CA Key"
  api-key: <your_api_key_here>

After you have added this secret (with the correct value) in the cert-manager folder, you should just need to invoke:

make cert-manager

gsaslis / home-on-k8s Goto Github PK

home-on-k8s's Introduction