Giter VIP home page Giter VIP logo

k3s-gitops-arm's Introduction

IMPORTANT NOTE

This repository is really out of date, I have moved onto Flux v2 and do not use Raspberry Pis anymore. This is now archived and I will leave it up for people to read.

k3s-gitops-arm

Kubernetes

Discord

Build a Kubernetes (k3s) cluster with RPis and utilize GitOps for managing cluster state. I would like to give a shout-out to k8s-gitops, the big brother of this repo, created by @billimek.

This repo uses a lot of multi-arch images provided by raspbernetes/multi-arch-images.

Note: A lot of files in this project have @CHANGEME comments, these are things that are specific to my set up that you may need to change.

Prerequisites

Hardware

  • 3x Raspberry Pi 4 (recommended 4GB RAM model)
  • 3x SD cards (recommended 32GB)
  • 3x USB 3.x flash drives (recommended for local storage)
  • A NFS server for storing persistent data (recommended for shared storage)

Software

Note: I use the fish shell for a lot of my commands. Some will work in Bash but others will not, see here for more information.

Directory topology

.
├── ./ansible        # Ansible playbook to run after the RPis have been flashed
├── ./deployments    # Flux will only scan and deploy from this directory
├── ./setup          # Setup of the cluster
├── ./secrets        # Scripts to generate secrets for Sealed Secrets
└── ./docs           # Documentation

Network topology

image

IP Function
192.168.1.1 Router (USG)
192.168.1.170 NFS Server
192.168.42.1/24 k3s cluster CIDR, VLAN 42
192.168.42.23 k3s master (k3s-master)
192.168.42.24 k3s worker (k3s-worker-a)
192.168.42.25 k3s worker (k3s-worker-b)

Let's get started

1. Flash SD Card with Ubuntu

See ubuntu.md

2. Provision RPis with Ansible

Ansible is a great automation tool and here I am using it to provision the RPis.

See ansible.md and review the files in the ansible folder.

3. Install k3s on your RPis using k3sup

k3sup is a neat tool provided by @alexellis that helps get your k3s cluster up and running quick.

For manual deployment see k3sup.md, and for an automated script see bootstrap-cluster.sh

4. Flux and Helm Operator

Helm is a package manager for Kubernetes.

Flux is the GitOps tool I've chosen to have this Git Repository manage my clusters state.

For manual deployment see helm-flux.md, and for an automated script see bootstrap-cluster.sh

Additional Components

Sealed Secrets

Sealed Secrets are a "one-way" encrypted Secret that can be created by anyone, but can only be decrypted by the controller running in the target cluster. The Sealed Secret is safe to share publicly, upload to git repositories, give to the NSA, etc. Once the Sealed Secret is safely uploaded to the target Kubernetes cluster, the sealed secrets controller will decrypt it and recover the original Secret.

See sealed-secrets.md and review the files in the secrets folder.

MetalLB

MetalLB is a load-balancer implementation for bare metal Kubernetes clusters, using standard routing protocols.

Review the file metallb.yaml

Cert Manager

Cert-Manager will automatically provision and manage TLS certificates in Kubernetes. In this setup I am using Cloudflare as the DNS challenge.

NGINX Ingress /engine x/

NGINX Ingress Controller is an Ingress controller that uses ConfigMap to store the NGINX configuration.

Review the file nginx-ingress.yaml

k3s-gitops-arm's People

Contributors

onedr0p avatar radum avatar stefancrain avatar

Stargazers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Watchers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Forkers

gtamang001 cedricmc shahbinoy dalekurt mariusrugan talmuth bjornmagnusson hyposcaler narimantos radum asosso sigtom vgiannoul haphan svefors mike-ensor andreaseger jahroots c-yco devopstoday11 anthonyneto feyobekius swipswaps zsiddiqi doytsujin cedlerouge

k3s-gitops-arm's Issues

add metallb to setup

Noticed the graphic included the use of metallb but none of the documentation mention it.

Ansible systemd config is not working

The following doesn't get applied for some reason...

- name: Start and enable timesyncd
  systemd:
    name: systemd-timesyncd
    state: started
    enabled: true

vagrant up fails missing example.config.yml

Looks like example.config.yml is not part of repo quite yet. Maybe the error should reference vars.example.yml

$ vagrant up
Vagrant failed to initialize at a very early stage:

There was an error loading a Vagrantfile. The file being loaded
and the error message are shown below. This is usually caused by
a syntax error.

Path: tests/vagrant/Vagrantfile
Line number: 0
Message: RuntimeError: Configuration file not found! Please copy example.config.yml to config.yml and try again.

Search

$ find . -iname "example.config.yml"
nada

Missing Namespace Manually Installing Flux

I ran into an issue with the manual instructions for installing Flux and Helm.

(Below, namespaces in the name of the equivalent folder to your deployments)
helm upgrade --install flux --values namespaces/flux/flux/flux-values.yaml --namespace flux fluxcd/flux

I had to manually kubectl apply -f ./flux/namespace.yaml and then the command would work.

Ubuntu 20 and Userland changes

Hello,

First of all, thank you very much for this repo, it has saved me lots of hours and pain by learning from example. Really thank you.

I wanted to point out first that this will not work for Ubuntu 20 as k3s has an https://github.com/rancher/k3s/issues/1712 for now.

And 2nd I wanted to ask if PI userland patch still works. There have been some changes in that repo 2 days ago and the patch is failing.

Was wondering if you know why and how to fix it.

fatal: [192.168.0.100]: FAILED! => changed=true
  cmd: |-
    set -o pipefail
    echo /opt/vc/lib | tee /etc/ld.so.conf.d/00-vmcs.conf
  delta: '0:00:00.006693'
  end: '2020-05-30 23:38:15.220359'
  msg: non-zero return code
  rc: 2
  start: '2020-05-30 23:38:15.213666'
  stderr: '/bin/sh: 1: set: Illegal option -o pipefail'
  stderr_lines: <omitted>
  stdout: ''
  stdout_lines: <omitted>
fatal: [192.168.0.101]: FAILED! => changed=true
  cmd: |-
    set -o pipefail
    echo /opt/vc/lib | tee /etc/ld.so.conf.d/00-vmcs.conf
  delta: '0:00:00.006634'
  end: '2020-05-30 23:38:15.312757'
  msg: non-zero return code
  rc: 2
  start: '2020-05-30 23:38:15.306123'
  stderr: '/bin/sh: 1: set: Illegal option -o pipefail'
  stderr_lines: <omitted>
  stdout: ''
  stdout_lines: <omitted>
fatal: [192.168.0.102]: FAILED! => changed=true
  cmd: |-
    set -o pipefail
    echo /opt/vc/lib | tee /etc/ld.so.conf.d/00-vmcs.conf
  delta: '0:00:00.006775'
  end: '2020-05-30 23:38:15.418153'
  msg: non-zero return code
  rc: 2
  start: '2020-05-30 23:38:15.411378'
  stderr: '/bin/sh: 1: set: Illegal option -o pipefail'
  stderr_lines: <omitted>
  stdout: ''
  stdout_lines: <omitted>

Conflicting CRD Manually Installing Helm Operator

Manually installing Helm Operator produces this conflicting CRD.

Error: rendered manifests contain a resource that already exists. Unable to continue with install: existing resource conflict: namespace: , name: helmreleases.helm.fluxcd.io, existing_kind: apiextensions.k8s.io/v1beta1, Kind=CustomResourceDefinition, new_kind: apiextensions.k8s.io/v1beta1, Kind=CustomResourceDefinition

Solution was to remove the createCRD: true from the flux-helm-operator-values.yaml

Flux Helm Chart has been Updated; Instructions do not work

Hi, first of all: thanks for the effort - helped a lot :)

I noticed, that the instructuion in docsflux-helm-operator.md are pulling the latest helm chart for flux.

This currently leads to a not starting flux container, as in flux helm-chart 1.3.0 an additional parameter (ssh-keygen-format) has been added, which is not present in flux 1.18.

Not working:

helm upgrade --install flux \
    --values deployments/flux/flux/flux-values.yaml \
    --namespace flux \
    fluxcd/flux

Working:

helm upgrade --install flux \
    --values deployments/flux/flux/flux-values.yaml \
    --version 1.2.0
    --namespace flux \
    fluxcd/flux

Use helm v3

Waiting on helm-operator to release a new version with this Helm 3 short. I can remove tiller once that is done.

pre-commit fails, cannot find node 

pre-commit run --all-files
[INFO] Initializing environment for git://github.com/pre-commit/pre-commit-hooks.
[INFO] Initializing environment for https://github.com/jorisroovers/gitlint.
[INFO] Initializing environment for https://github.com/openstack/bashate.
[INFO] Initializing environment for https://github.com/mattlqx/pre-commit-sign.
[INFO] Initializing environment for https://github.com/Lucas-C/pre-commit-hooks.
[INFO] Initializing environment for https://github.com/jumanjihouse/pre-commit-hooks.
[INFO] Initializing environment for https://github.com/ansible/ansible-lint.git.
[INFO] Initializing environment for https://github.com/igorshubovych/markdownlint-cli.
[INFO] Installing environment for git://github.com/pre-commit/pre-commit-hooks.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
[INFO] Installing environment for https://github.com/openstack/bashate.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
[INFO] Installing environment for https://github.com/Lucas-C/pre-commit-hooks.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
[INFO] Installing environment for https://github.com/ansible/ansible-lint.git.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
[INFO] Installing environment for https://github.com/igorshubovych/markdownlint-cli.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
An unexpected error has occurred: CalledProcessError: Command: ('/usr/local/Cellar/pre-commit/1.20.0/libexec/bin/python3.7', '-mnodeenv', '--prebuilt', '--clean-src', '/Users/devin/.cache/pre-commit/repotmwccurm/node_env-system', '-n', 'system')
Return code: 1
Expected return code: 0
Output: (none)
Errors:
    Traceback (most recent call last):
      File "/usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py", line 193, in _run_module_as_main
        "__main__", mod_spec)
      File "/usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py", line 85, in _run_code
        exec(code, run_globals)
      File "/usr/local/Cellar/pre-commit/1.20.0/libexec/lib/python3.7/site-packages/nodeenv.py", line 1480, in <module>
        main()
      File "/usr/local/Cellar/pre-commit/1.20.0/libexec/lib/python3.7/site-packages/nodeenv.py", line 1076, in main
        create_environment(env_dir, opt)
      File "/usr/local/Cellar/pre-commit/1.20.0/libexec/lib/python3.7/site-packages/nodeenv.py", line 913, in create_environment
        install_activate(env_dir, opt)
      File "/usr/local/Cellar/pre-commit/1.20.0/libexec/lib/python3.7/site-packages/nodeenv.py", line 845, in install_activate
        assert shim_node, "Did not find nodejs or node system executable"
    AssertionError: Did not find nodejs or node system executable

Check the log at /Users/devin/.cache/pre-commit/pre-commit.log

Node is here :weirdface:

❯ node --version
v13.2.0

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.