Sometimes there is a race condition between Flux deploying and the correct sealed-secrets being applied.

One solution could be to explicitly apply all sealed-secrets from the pipeline, so that the correct values are in place before Flux comes along.

Steps:

• Add kubeconfig (ENCRYPTED!) to the cluster
• Make the pipeline use this kubeconfig file
• Add a step in the sealed-secrets workflow to apply all sealed-secrets

Details

Describe the solution you'd like:

Replace the current Traefik ingress controller with Nginx

Currently, I have a system for variables in place, but I don't use those variables enough in my secrets yet.

Currently there is overlap between the setup script and the fact that there is Ansible code present. Therefore the setup script should somehow converge into an Ansible role

Errors were reported while checking the availability of links.

📝 Summary
---------------------
🔍 Total..........115
✅ Successful.....114
⏳ Timeouts.........0
🔀 Redirected.......0
👻 Excluded.........0
🚫 Errors...........1

Errors in networking/dns/index.html
✗ https://opnsense.org/ (error sending request for url (https://opnsense.org/): error trying to connect: dns error: failed to lookup address information: Temporary failure in name resolution)

Full Github Actions output

Can you explain a bit on the server side options of NFS ?
Are you using root_squash ?
If yes then how are you managing all the user permissions ? A lot of containers/pods chown to different users:groups and it becomes a bit hectic to keep track of all of them.

Details

Describe the solution you'd like:

I should be using the full device UUID path when consuming drives in rook-ceph

After Kured drained and rebooted my worker nodes, zigbee2mqtt would no longer actually perform any commands even though it seemed to be up and running.

The error message was as follows:

Error while starting zigbee-herdsman
Error: Failed to connect to adapter (Error: SRSP - SYS - ping after 6000ms)

Resolution:

• Scale down the zigbee2mqtt deployment to 0 replicas
• Unplug the CC2652R adapter and replug it
• Scale up the zigbee2mqtt deployment to 1 replicas

zigbee-herdsman now succesfully started and the zigbee network was now reachable again.

Since having the lights not respond once every two weeks (current Kured schedule) is not acceptable, I've decided to remove Kured from the cluster for now.

Currently there is nothing in place for collecting metrics, and presenting them.

Add something like a Prometheus / Grafana stack for this.

Errors were reported while checking the availability of links.

📝 Summary
---------------------
🔍 Total..........132
✅ Successful.....120
⏳ Timeouts.........0
🔀 Redirected.......0
👻 Excluded.........0
🚫 Errors..........12

Errors in index.html
✗ https://fonts.gstatic.com/ (HTTP status client error (404 Not Found) for url (https://fonts.gstatic.com/))

Errors in networking/dns/index.html
✗ https://fonts.gstatic.com/ (HTTP status client error (404 Not Found) for url (https://fonts.gstatic.com/))

Errors in storage/index.html
✗ https://fonts.gstatic.com/ (HTTP status client error (404 Not Found) for url (https://fonts.gstatic.com/))

Errors in storage/backups/index.html
✗ https://fonts.gstatic.com/ (HTTP status client error (404 Not Found) for url (https://fonts.gstatic.com/))

Errors in networking/multus/index.html
✗ https://fonts.gstatic.com/ (HTTP status client error (404 Not Found) for url (https://fonts.gstatic.com/))

Errors in networking/podgateway/index.html
✗ https://fonts.gstatic.com/ (HTTP status client error (404 Not Found) for url (https://fonts.gstatic.com/))

Errors in 404.html
✗ https://fonts.gstatic.com/ (HTTP status client error (404 Not Found) for url (https://fonts.gstatic.com/))

Errors in networking/index.html
✗ https://fonts.gstatic.com/ (HTTP status client error (404 Not Found) for url (https://fonts.gstatic.com/))

Errors in gitops/index.html
✗ https://fonts.gstatic.com/ (HTTP status client error (404 Not Found) for url (https://fonts.gstatic.com/))

Errors in home/cluster_overview/index.html
✗ https://fonts.gstatic.com/ (HTTP status client error (404 Not Found) for url (https://fonts.gstatic.com/))

Errors in home/tools/index.html
✗ https://fonts.gstatic.com/ (HTTP status client error (404 Not Found) for url (https://fonts.gstatic.com/))

Errors in home/repo_structure/index.html
✗ https://fonts.gstatic.com/ (HTTP status client error (404 Not Found) for url (https://fonts.gstatic.com/))

Full Github Actions output

Summary

Errors per input

Errors in ./docs/networking/dns.md

• file:///home/runner/work/home-ops/home-ops/_assets/images/dns_graph.svg: Failed: Cannot find file

Errors in ./docs/home/cluster_overview.md

• file:///home/runner/work/home-ops/home-ops/storage: Failed: Cannot find file
• file:///home/runner/work/home-ops/home-ops/networking: Failed: Cannot find file

Full Github Actions output

Currently all backups are on the local network. For additional security I want to add a cloud backup of this data in the mix.

Should look at how @onedr0p does this with rclone to Backblaze B2
https://github.com/onedr0p/k3s-gitops/tree/master/deployments/default/rclone

Errors were reported while checking the availability of links.

📝 Summary
---------------------
🔍 Total..........132
✅ Successful.....120
⏳ Timeouts.........0
🔀 Redirected.......0
👻 Excluded.........0
🚫 Errors..........12

Errors in networking/index.html
✗ https://fonts.gstatic.com/ (HTTP status client error (404 Not Found) for url (https://fonts.gstatic.com/))

Errors in home/tools/index.html
✗ https://fonts.gstatic.com/ (HTTP status client error (404 Not Found) for url (https://fonts.gstatic.com/))

Errors in storage/index.html
✗ https://fonts.gstatic.com/ (HTTP status client error (404 Not Found) for url (https://fonts.gstatic.com/))

Errors in storage/backups/index.html
✗ https://fonts.gstatic.com/ (HTTP status client error (404 Not Found) for url (https://fonts.gstatic.com/))

Errors in gitops/index.html
✗ https://fonts.gstatic.com/ (HTTP status client error (404 Not Found) for url (https://fonts.gstatic.com/))

Errors in networking/dns/index.html
✗ https://fonts.gstatic.com/ (HTTP status client error (404 Not Found) for url (https://fonts.gstatic.com/))

Errors in networking/multus/index.html
✗ https://fonts.gstatic.com/ (HTTP status client error (404 Not Found) for url (https://fonts.gstatic.com/))

Errors in networking/podgateway/index.html
✗ https://fonts.gstatic.com/ (HTTP status client error (404 Not Found) for url (https://fonts.gstatic.com/))

Errors in 404.html
✗ https://fonts.gstatic.com/ (HTTP status client error (404 Not Found) for url (https://fonts.gstatic.com/))

Errors in home/repo_structure/index.html
✗ https://fonts.gstatic.com/ (HTTP status client error (404 Not Found) for url (https://fonts.gstatic.com/))

Errors in index.html
✗ https://fonts.gstatic.com/ (HTTP status client error (404 Not Found) for url (https://fonts.gstatic.com/))

Errors in home/cluster_overview/index.html
✗ https://fonts.gstatic.com/ (HTTP status client error (404 Not Found) for url (https://fonts.gstatic.com/))

Full Github Actions output

K3s seems to be a more lightweight approach to initialize a cluster. Should try implementing this during the next cluster iteration

Details

Describe the solution you'd like:

Currently there is some hackery going on to download/update/make available a template chart based on the KaH common library.
Now that there is an upstream version of that (k8s-at-home/charts#1501), use that.

Details

Describe the solution you'd like:

A lot of the Renovate config snippets in this repo can be broken out into a separate repository for improved reusability.

Details

What steps did you take and what happened:

I have a few CronJobs that run in my cluster periodically. They run fine, but randomly throw an error stating that a volume was not registered.

What did you expect to happen:

They run fine and don't raise any errors.

Additional Information:

The error is harmless otherwise, but wanted to document it here otherwise.

Upstream issue: kubernetes/kubernetes#105204

Summary

Errors per input

Errors in ./docs/networking/dns.md

• file:///home/runner/work/home-ops/home-ops/_assets/images/dns_graph.svg: Failed: Cannot find file

Errors in ./docs/home/cluster_overview.md

• file:///home/runner/work/home-ops/home-ops/networking: Failed: Cannot find file
• file:///home/runner/work/home-ops/home-ops/storage: Failed: Cannot find file

Full Github Actions output

Don't forget to change your renovate config to catch the repo rename.

Details

Describe the solution you'd like:

Do something similar to the home-assistant deployment:

• Split into two Kustomizations: database and application (with application depending on database)
• Do a pg_dump from the existing database and pg_restore into the new one
• Remove the Kubegres manifest from dsmr-reader

It turns out that Calico has built-in support for BGP. This would make running MetalLB no longer necessary, as that's basically all I'm using it for.

Links:

• Discord discussion

Summary

Errors per input

Errors in ./docs/home/cluster_overview.md

• file:///home/runner/work/home-ops/home-ops/storage: Failed: Cannot find file
• file:///home/runner/work/home-ops/home-ops/networking: Failed: Cannot find file

Errors in ./docs/networking/dns.md

• file:///home/runner/work/home-ops/home-ops/_assets/images/dns_graph.svg: Failed: Cannot find file

Full Github Actions output

Context:
Currently my rook-ceph stuff lives in the cluster-apps Kustomization. I should move these to the cluster-requirements Kustomization so that any errors in apps don't potentially nuke the cluster storage.

Tasks

• Move rook-ceph folder from apps to requirements
• Set prune: false on cluster-requirements Kustomization
• Check if there are any CRD's that need to be manually managed

Implement kustomize.yaml files to manage dependency ordering.

Resources:

• https://toolkit.fluxcd.io/components/kustomize/api/

Summary

Errors per input

Errors in ./docs/home/cluster_overview.md

• file:///home/runner/work/home-ops/home-ops/storage: Failed: Cannot find file
• file:///home/runner/work/home-ops/home-ops/networking: Failed: Cannot find file

Errors in ./docs/networking/dns.md

• file:///home/runner/work/home-ops/home-ops/_assets/images/dns_graph.svg: Failed: Cannot find file

Full Github Actions output

Benji seems to be a lot more light-weight, and covers my use case: backup PVC content to an NFS share.

Reference materials:

• https://github.com/elemental-lf/benji
• https://github.com/toboshii/home-cluster/tree/main/cluster/apps/backup-system

Currently I have a few places where a hpa is used to enforce min/max replicas. This is less then ideal for a number of reasons:

• if the new replicacount exceeds the specified maximum, it will first scale up and then scale down, causing a temporary situation max+1 situation.
• not all charts support hpa out of the box

In order to remedy this, I should probably look into something like Open Policy Agent

Links:

• https://www.magalix.com/blog/introducing-policy-as-code-the-open-policy-agent-opa
• https://www.openpolicyagent.org/docs/latest/kubernetes-tutorial/#3-deploy-opa-on-top-of-kubernetes

Currently there are a number of services with seperate manifest files.These should be migrated to Helm charts.

Either in my own charts repo, or preferably in @billimek's charts repo

Create chart

• airconnect
• dsmr-reader (k8s-at-home/charts#426)
• emby
• nut-upsd
• samba

Replace with alternative

• mosquitto

Context:
Currently I mostly rely on service built-in authentication mechanisms. It would be better / easier to set up a separate authentication mechanism so that I can centralise my user accounts, and possibly even implement SSO.

Related issues:

• billimek/k8s-gitops#36

Details

Describe the solution you'd like:

Now that I only have one cluster to manage it makes sense to roll the global folder back in to the cluster-0 folder.

Additional Information:

Depends on #2514

Details

Describe the solution you'd like:

I currently use iCloud Drive to store my personal documents archive. I want to investigate moving that over to paperless-ngx

Additional Information:

Application repo: https://github.com/paperless-ngx/paperless-ngx

Due to the old cluster going down while I was still migrating stuff into this repo, not everything is in place yet.

The following services still need to be migrated:

Applications:

• blocky
• Node-RED ("brains")
• calibre-web
• dsmr-reader
• LazyLibrarian
• Radarr
• ssh server
• zigbee2mqtt

Scripts:

• Email backup script
• Series cleanup script

Details

Describe the solution you'd like:

Instead of adding the same custom annotation anywhere, implement a mutating policy with Kyverno.

The fluxv2 multitenancy example folder structure looks pretty neat. Investigate if refactoring is worth it.

Resources:

• https://github.com/fluxcd/flux2-multi-tenancy

Details

Describe the solution you'd like:

When moving away from Sidero PXE booting I need a way to (securely) store the machineConfig definitions in this git repo.

Details

Describe the solution you'd like:

Remove the Kubegres Kustomization from the cluster

Additional Information:

Depends on #2492

Summary

Errors per input

Errors in ./docs/networking/dns.md

• file:///home/runner/work/home-ops/home-ops/_assets/images/dns_graph.svg: Failed: Cannot find file

Errors in ./docs/home/cluster_overview.md

• file:///home/runner/work/home-ops/home-ops/storage: Failed: Cannot find file
• file:///home/runner/work/home-ops/home-ops/networking: Failed: Cannot find file

Full Github Actions output

Details

Describe the solution you'd like:

Deploy https://github.com/GilbN/theme.park/ in the cluster to add themes to apps.

Relevant links

• https://github.com/Truxnell/container-images/tree/main/apps/theme-park
• https://github.com/Truxnell/home-cluster/blob/main/k8s/manifests/services/theme-park/helmrelease.yaml
• https://github.com/onedr0p/home-ops/tree/main/cluster/apps/default/theme-park

Additional Information:

Run on port 8080 because the container runs as non-root

Depends on #2494

Errors were reported while checking the availability of links.

📝 Summary
---------------------
🔍 Total..........115
✅ Successful.....114
⏳ Timeouts.........0
🔀 Redirected.......0
👻 Excluded.........0
🚫 Errors...........1

Errors in networking/dns/index.html
✗ https://opnsense.org/ (error sending request for url (https://opnsense.org/): error trying to connect: dns error: failed to lookup address information: Temporary failure in name resolution)

Full Github Actions output

Hi,
i'd like to use your work to create my first home cluster (zero experience on that).
I first tried k8s-at-home/template-cluster-k3s, i followed all the steps and it worked but i'd like to use yours so i'm "yelling" for a little help. Actually is a big please.

Hardware that i have:

• Mikrotik router which seems able to work with CoreDNS
• TrueNas for storage
• XCP-ng Hypervisor for VMs

Things that confuses me:

• i don't see any ansible so how can i deploy cluster (seems to be in .gitignore) ?
• what is "siderio", it seems is another cluster that deploys "cluster-o" ?
• is a must to have multiple nodes for this cluster to work ?

Could you lease help me with some hints to deploy the cluster, set networks, storages, lb, dns ? :)

Thank you

Details

Describe the solution you'd like:

With Traefik there was a middleware in place to whitelist internal Ingress objects to RFC1918 IP ranges.
Add nginx.ingress.kubernetes.io/whitelist-source-range to these Ingresses to reimplement this whitelist.

Additional Information:

https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#whitelist-source-range

Depends on #2502

Details

Describe the solution you'd like:

Add theme.park themes to applications that support it:

• nzbget
• qbittorrent
• radarr
• sonarr
• lidarr
• readarr
• prowlarr
• calibre-web

Additional Information:

See https://github.com/GilbN/theme.park/ for available themes / apps

Currently I have a single Runner (actions-runner-controller) for this repo. Maybe I should modify this to a RunnerDeployment so I can have two parallel runners.

Summary

Errors per input

Errors in ./docs/networking/dns.md

• file:///home/runner/work/home-ops/home-ops/_assets/images/dns_graph.svg: Failed: Cannot find file

Errors in ./docs/home/cluster_overview.md

• file:///home/runner/work/home-ops/home-ops/storage: Failed: Cannot find file
• file:///home/runner/work/home-ops/home-ops/networking: Failed: Cannot find file

Full Github Actions output

Because of the impending rate limits on Docker Hub I need to set up a pull-through cache.

Most likely solution: run the Nexus as a docker container on my NAS and add that as a mirror repository to the k3s config

• General: Landing page
• General: Add shields to landing page
• General: Hardware
• General: Networking
• General: Storage
• General: Storage backups
• Automation: GitOps / Flux
• Automation: Renovate
• Networking: DNS
• Networking: Multus
• Networking: PodGateway

Details

Describe the solution you'd like:

Rename the root k8s folder to kubernetes. This means that a lot of Flux resources need to be updated as well, and potentially CI configuration.

Anything else you would like to add:

Additional Information:

There is an error with this repository's Renovate configuration that needs to be fixed. As a precaution, Renovate will stop PRs until it is resolved.

Location: .github/renovate.json5
Error type: The renovate configuration file contains some invalid settings
Message: Invalid configuration option: flux

Currently some critical (prometheus, certmanager) CRD's are installed by Helm charts. In order to prevent chicken/egg situations when (re)deploying the cluster I should probably separate these and add them to the bootstrap script/role.