locmai / humble Goto Github PK

View Code? Open in Web Editor NEW

75.0 3.0 14.0 11.67 MB

Just a humble project

Home Page: https://humble.maibaloc.com

License: GNU General Public License v3.0

Makefile 14.08% Shell 6.29% Jinja 18.77% Dockerfile 0.54% HCL 24.38% Nix 7.62% Python 28.31%

terraform argocd infrastructure-as-code kubernetes homelab

humble's Introduction

Humble Project

Overview

Humble is an open-source project that seeks to set up a ready-to-go environment with operating services using modern infrastructure as code with GitOps driven.

Let's say you have a couple of physical servers at home, and you would love to deploy your applications or services on them with Kubernetes platform. This project intends to automate all necessary components and boot them up from scratch by just a single command from your laptop or workstation.

Core Features

Fully automated with infrastructure as code
Modularized components stand on layered architecture
Versioned and declarative infrastructure on top of GitOps
Supports data-driven deployment for multiple environments

Documentation

See the docs for detailed information on the architecture, installation and use of the platform.

For getting started:

Try it out locally without any hardware.
Deploy on real hardware for production workload.

Acknowledgements

I hacked a lot of great works from khuedoan/homelab.
Awesome managed services from Cloudflare, GitHub, and Tailscale: DNS, Pages and Tunnel.
All open-source projects used in this project.

Contacts

Email me at [email protected] or create an issue on the GitHub repository.

humble's People

Contributors

Stargazers

Watchers

Forkers

anhtran1610 tdien105 dtrh95 draconis2941 imerkle bridgecrew-perf4 gorathon kre80r cameronraysmith deltacodepl djonko getdatasurge carstenthom sokoobo

humble's Issues

Dependency Dashboard

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Repository problems

Renovate tried to run on this repository, but found these problems.

WARN: Found renovate config warnings
WARN: Invalid regex manager registryUrl
WARN: Package lookup failures

Edited/Blocked

These updates have been manually edited so Renovate will no longer make changes. To discard all commits and start over, click on a checkbox.

chore(deps): update helm release app-template to v3

Warning

Renovate failed to look up the following dependencies: Failed to look up helm package glance, Failed to look up helm package secret-generator.

Files affected: apps/templates/glance.yaml, platform/templates/global-secrets.yaml

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

chore(deps): update all non-major dependencies (actions/checkout, argo-cd, cassandra, cert-manager, common, coroot, ingress-nginx, nginx, opentelemetry-collector, reloader, renovatebot/github-action)
chore(deps): update helm release kube-prometheus-stack to v58

Detected dependencies

docker-compose

metal/roles/pxe-server/files/docker-compose.yml

dockerfile

Dockerfile.tools

metal/roles/pxe-server/files/dnsmasq/Dockerfile

alpine 3.19

metal/roles/pxe-server/files/http/Dockerfile

nginx 1.25.5-alpine

github-actions

.github/workflows/pr-checks.yaml

amannn/action-semantic-pull-request v5

actions/checkout v4

azure/setup-helm v4

actions/setup-python v5

helm/chart-testing-action v2.6.1

.github/workflows/release-please.yaml

google-github-actions/release-please-action v4

.github/workflows/renovate.yaml

actions/checkout v4.1.1

renovatebot/github-action v40.1.10

helmv3

bootstrap/argocd/Chart.yaml

argo-cd 6.7.12

platform/Chart.yaml

common 2.19.1

pip_requirements

docs/requirements.txt

mergedeep ==1.3.4

python-dateutil ==2.9.0.post0

PyYAML ==6.0.1

pyyaml_env_tag ==0.1

six ==1.16.0

terraform

global/_modules/external-dns/providers.tf

global/_modules/external-dns/versions.tf

cloudflare ~> 4.30.0

http ~> 3.4.0

kubernetes ~> 2.29.0

hashicorp/terraform >= 1.0.0

terragrunt

global/prod/terragrunt.hcl

global/stag/terragrunt.hcl

regex

apps/templates/excalidraw.yaml

excalidraw 0.0.3

apps/templates/glance.yaml

glance 0.1.0

apps/templates/homepage.yaml

app-template 2.6.0

apps/templates/jellyfin.yaml

app-template 2.6.0

apps/templates/speedtest.yaml

app-template 2.6.0

global/templates/cloudflared.yaml

cloudflared 0.4.3

global/templates/external-dns.yaml

external-dns 1.14.4

platform/templates/cassandra.yaml

cassandra 11.1.0

platform/templates/coroot.yaml

coroot 0.10.4

platform/templates/dexidp.yaml

dex 0.17.1

platform/templates/external-secrets.yaml

external-secrets 0.9.16

platform/templates/global-secrets.yaml

secret-generator 0.1.10

platform/templates/kanidm.yaml

app-template 2.6.0

platform/templates/reloader.yaml

reloader 1.0.80

platform/templates/temporal.yaml

temporal 0.32.0

system/templates/cert-manager.yaml

cert-manager v1.14.4

system/templates/cilium-monitor.yaml

cilium-monitor 0.0.3

system/templates/ingress-nginx.yaml

ingress-nginx 4.10.0

system/templates/loki.yaml

loki-stack 2.10.2

system/templates/mimir.yaml

mimir-distributed 5.3.0

system/templates/monitoring.yaml

kube-prometheus-stack 57.2.0

system/templates/opentelemetry.yaml

opentelemetry-collector 0.87.2

system/templates/tracing.yaml

jaeger 0.51.6

apps/templates/excalidraw.yaml

excalidraw 0.0.3

apps/templates/glance.yaml

glance 0.1.0

apps/templates/homepage.yaml

app-template 2.6.0

apps/templates/jellyfin.yaml

app-template 2.6.0

apps/templates/speedtest.yaml

app-template 2.6.0

global/templates/cloudflared.yaml

cloudflared 0.4.3

global/templates/external-dns.yaml

external-dns 1.14.4

platform/templates/cassandra.yaml

cassandra 11.1.0

platform/templates/coroot.yaml

coroot 0.10.4

platform/templates/dexidp.yaml

dex 0.17.1

platform/templates/external-secrets.yaml

external-secrets 0.9.16

platform/templates/global-secrets.yaml

secret-generator 0.1.10

platform/templates/kanidm.yaml

app-template 2.6.0

platform/templates/reloader.yaml

reloader 1.0.80

platform/templates/temporal.yaml

temporal 0.32.0

system/templates/cert-manager.yaml

cert-manager v1.14.4

system/templates/cilium-monitor.yaml

cilium-monitor 0.0.3

system/templates/ingress-nginx.yaml

ingress-nginx 4.10.0

system/templates/loki.yaml

loki-stack 2.10.2

system/templates/mimir.yaml

mimir-distributed 5.3.0

system/templates/monitoring.yaml

kube-prometheus-stack 57.2.0

system/templates/opentelemetry.yaml

opentelemetry-collector 0.87.2

system/templates/tracing.yaml

jaeger 0.51.6

Cilium installation v2

Just an issue to mark this done - as hacked from khuedoan again

Setup lint GH Actions pipeline

Lints:

helm
terraform
ansible
yaml

Tests:

PRs check:

Naming

Add Cert-Manager

Letting CloudFlare handles all the HTTPS requests doesn't meet all requirements for some component (like Keycloak)

Need an instance of cert-manager for dynamically self-signing the certs.

Cilium and Hubble setup

Replace the default CNI (Flannel) with Cilium to provide networking, observability, and security solution with the eBPF-based dataplane.

Replace the default CNI installation with Cilium
Enable observability mode - get default metrics into Prometheus
Test ingress

Setup wg-access-server

Deploy this server on layer 1 node 01: https://github.com/Place1/wg-access-server

AC1: Could access the VPN via this server.

Grafana Sidecar Dashboard

Setup sidecar mechanism to dynamically inject the dashboards into the main grafana

Automatically initialize Vault

From provisioning flow, after Vault installation, we will need to initialize Vault with root token, shared keys and unseal it. Then enable Kubernetes authentication and feed the initial secret set into Vault.

Write a script that:

Initialize Vault with root token and shared keys
Unseal Vault with the generated root token and shared keys
Enable Kubernetes authentication
Feed secrets into Vault

Refs:

hvac Vault client
kubernetes kubernetes client

Evaluate offsite storage

A few options:

https://www.gluster.org/
backblaze
Azure cold storage
S3 Glacier
https://www.oracle.com/cloud/storage/archive-storage/

Setup Cortex

Deploy Cortex: https://github.com/locmai/humble/blob/main/apps/templates/cortex.yaml

Items:

Deploy Vault Operator

Update all PR checks to GitHub Actions

Build the Sitespeed daily job for blog.locmai.dev

Add Git Webhook for ArgoCD

Using this provider: https://registry.terraform.io/providers/integrations/github/latest/docs
Configure the webhook to notfiy ArgoCD after every push. No share secret required.

Configure Renovate Bot for Humble and Yuta

Deploy staging environment

Implement Excalidraw Helm chart to have share/collaborate feature

We now have the base chart at https://github.com/locmai/charts/tree/main/charts/excalidraw - a few extra steps required to have the collaboration feature:

Follow the PoC with docker-compose here https://gitlab.open-mind.space/pandel/excalidraw-docker
The PoC seems referencing to the work at https://github.com/NMinhNguyen/excalidraw-json which has no Dockerfile yet.
Spin up the excalidraw-room, excalidraw-json, and Minio then route ingress accordingly.

Setup external-secrets

This issue will be started after #35 with the initial secrets.

Setup external-secrets with helm
Hook external-secrets with Vault
Write all ExternalSecrets

Provide document on secret management

Provide a document for the works in #35 and #37

Evaluate and test PostgreSQL backup

Spin up a minio and a PostgreSQL instance.
Add backup workflow.
Test by feeding some data into the PostgreSQL instance, then run the backup workflow and store the current data to minio, then nuke the database and restore.
Also have a background service for pinging and querying data from the database to check the downtime/data loss

Migrate documents to mkdocs

Upsize local storage

Deploy saleor

Complete the Saleor charts at:

Then deploy it with the ingress to https://saleor.maibaloc.com

K8up installation

Deploy Ory Kratos

deploy mailhog as the development SMTP server, find the chart here https://artifacthub.io/packages/helm/codecentric/mailhog
run kratos migration against the main postgresql
deploy the ory kratos chart (https://github.com/locmai/ory-k8s)

Static security check

Provide a simple static security scan/check for the following cases:

Container Image Scanning
Kubernetes manifests
Node/OS scan? CVE dashboard?

Note: Policy enforcement like OPA should be in another issue considered the size of the implementation for it.

Database backup for Dendrite

Action Required: Fix Renovate Configuration

There is an error with this repository's Renovate configuration that needs to be fixed. As a precaution, Renovate will stop PRs until it is resolved.

Location: .*repoURL: (?<registryUrl>(?!{{ .Values.global.repoURL }}|'https://github.com/locmai/humble-grafana-dashboards.git').*?)\n
Error type: Invalid regular expression: .*repoURL: (?(?!{{ .Values.global.repoURL }}|'https://github.com/locmai/humble-grafana-dashboards.git').*?)\n

Deploy backstage app

Prepare v0.1.0 release

Prepare to cut the release before the Humble Revamp started.

Revert all test code.
Revert back all apps to stable version.
Release doc

Deploy Excalidraw

Simply add excalidraw app deployment. The chart is set here https://github.com/locmai/charts/tree/main/charts/excalidraw

Evaluate kubernetes test executor framework

vault version

Hi, I noticed that you are using bank-vaults operator. Do you know what version of vault is the operator using?
I noticed that hashicorp has released vault version 1.12.x and somewhere in the bank-vaults config it mentioned vault 1.6.2 I wonder if that is the image version they use or an actual vault version number.

Evaluate and re-deploy Istio

Evaluate Istio
Deploy back Jaeger, OpenTelemetry Collector and microservice-demo
Testing and tuning

Setup system-upgrade-controller

Add a set of system maintainers/house cleaners:

Kured to perform safe reboot on the nodes
system-upgrade-controller
Descheduler for house cleaning

Consider more works for the future:

Rennovate bot

Add LXD container as GitHub Action self-hosted agent

Init a LXD container - ubuntu
Connect to GitHub as a self-hosted
Add a test job

Migrate from Prometheus to Mimir

Setup a minimal Mimir
Ensure the metrics flow into Mimir
Setup data sources with grafana.maibaloc.com

~~Switch Prometheus to agent mode and clean up persistent storage~~ move this to a new ticket

Deploy Tailscale

Build a simple PoC over any cloud provider
Codify the setup and implement test

Evaluate chaos testing framework

Dependency Dashboard

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Repository problems

These problems occurred while renovating this repository.

WARN: Invalid regex manager registryUrl

Edited/Blocked

These updates have been manually edited so Renovate will no longer make changes. To discard all commits and start over, click on a checkbox.

chore(deps): update all non-major dependencies (Markdown, alpine, argo-cd, argo-events, argo-workflows, authentik, base, cert-manager, cloudflare, cloudflared, colorama, consul, cortex, descheduler, etcd, external-dns, external-secrets, gateway, gitea, http, ingress-nginx, istiod, kube-prometheus-stack, kubernetes, kured, loki-stack, longhorn, memcached, metallb, mimir-distributed, minio, nginx, opentelemetry-collector, renovatebot/github-action, rich)

Ignored or Blocked

These are blocked by an existing closed PR and will not be recreated unless you click a checkbox below.

chore(deps): update actions/checkout action to v3

Detected dependencies

ansible

platform/kustomization/tekton/tasks/goodbye.yaml

ubuntu no version found

platform/kustomization/tekton/tasks/helloworld.yaml

docker/whalesay latest

docker-compose

metal/roles/pxe-server/files/docker-compose.yml

dockerfile

metal/roles/pxe-server/files/dhcp/Dockerfile

alpine 20220715

metal/roles/pxe-server/files/http/Dockerfile

nginx 1.22.0-alpine

metal/roles/pxe-server/files/tftp/Dockerfile

alpine 20220715

platform/files/Dockerfile

python 3.10-slim-buster

tools/Dockerfile

archlinux no version found

github-actions

.github/workflows/conventional-pr.yaml

amannn/action-semantic-pull-request v4

.github/workflows/renovate.yaml

actions/checkout v2.0.0

renovatebot/github-action v32.118.0

helm-values

bootstrap/argocd/values.yaml

alpine 3.15

helmv3

bootstrap/argocd/Chart.yaml

argo-cd 4.6.1

pip_requirements

docs/requirements.txt

click ==8.1.3

ghp-import ==2.1.0

importlib-metadata ==4.12.0

Jinja2 ==3.1.2

Markdown ==3.3.7

MarkupSafe ==2.1.1

mergedeep ==1.3.4

mkdocs ==1.3.0

mkdocs-material ==8.3.9

mkdocs-material-extensions ==1.0.3

packaging ==21.3

Pygments ==2.12.0

pymdown-extensions ==9.5

pyparsing ==3.0.9

python-dateutil ==2.8.2

PyYAML ==6.0

pyyaml_env_tag ==0.1

six ==1.16.0

watchdog ==2.1.9

zipp ==3.8.1

tools/requirements.txt

colorama ==0.4.4

commonmark ==0.9.1

Pygments ==2.12.0

rich ==12.4.1

regex

apps/templates/consul.yaml

consul 0.44.0

apps/templates/cortex.yaml

cortex 1.4.0

apps/templates/dashy-landscape.yaml

dashy 0.0.2

apps/templates/dendrite.yaml

dendrite 6.0.0

apps/templates/element.yaml

elementweb 0.0.3

apps/templates/filebrowser.yaml

filebrowser 0.0.2

apps/templates/memcached.yaml

memcached 6.0.18

apps/templates/microservices-demo.yaml

microservices-demo 0.0.1

apps/templates/nfs.yaml

nfs-server-provisioner 1.4.0

apps/templates/yutaops.yaml

yuta 0.0.2

bootstrap/root/templates/apps.yaml

undefined {{ .Values.apps.targetRevision }}

bootstrap/root/templates/bootstrap.yaml

undefined {{ .Values.global.targetRevision }}

bootstrap/root/templates/global.yaml

undefined {{ .Values.global.targetRevision }}

bootstrap/root/templates/platform.yaml

undefined {{ .Values.platform.targetRevision }}

bootstrap/root/templates/system.yaml

undefined {{ .Values.system.targetRevision }}

global/templates/cloudflared.yaml

cloudflared 0.3.3

global/templates/external-dns.yaml

external-dns 6.4.1

platform/templates/argo-events.yaml

argo-events 2.0.0

platform/templates/argo-workflow.yaml

argo-workflows 0.15.0

platform/templates/authentik.yaml

authentik 2022.4.3

platform/templates/etcd.yaml

etcd 8.1.3

platform/templates/external-secrets.yaml

external-secrets 0.5.3

platform/templates/gitea.yaml

gitea 5.0.8

platform/templates/kustom-init-secrets.yaml

undefined main

platform/templates/kustom-tekton.yaml

undefined {{ .Values.tekton.targetRevision }}

platform/templates/mimir.yaml

mimir-distributed 2.0.14

platform/templates/minio.yaml

minio 11.5.1

platform/templates/secret-generator.yaml

undefined {{ .Values.secretGenerator.targetRevision }}

platform/templates/trow.yaml

trow 0.3.5

system/templates/cert-manager.yaml

cert-manager v1.8.0

system/templates/dashboard.yaml

undefined {{ .Values.dashboard.targetRevision }}

system/templates/descheduler.yaml

descheduler 0.23.2

system/templates/ingress-nginx.yaml

ingress-nginx 4.1.1

system/templates/istio-base.yaml

base 1.13.4

system/templates/istio-gateway.yaml

gateway 1.13.4

system/templates/istio-istiod.yaml

istiod 1.13.4

system/templates/kured.yaml

kured 2.14.1

system/templates/kustom-system-upgrade.yaml

undefined {{ .Values.systemUpgrade.targetRevision }}

system/templates/loki.yaml

loki-stack 2.6.4

system/templates/longhorn.yaml

longhorn 1.2.4

system/templates/metallb.yaml

metallb 0.12.1

system/templates/monitoring.yaml

kube-prometheus-stack 37.0.0

system/templates/opentelemetry.yaml

opentelemetry-collector 0.17.0

system/templates/tracing.yaml

jaeger 0.51.6

terraform

global/_modules/external-dns/main.tf

undefined no version found

undefined no version found

undefined no version found

undefined no version found

undefined no version found

undefined no version found

undefined no version found

undefined no version found

global/_modules/external-dns/providers.tf

cloudflare no version found

global/_modules/external-dns/versions.tf

hashicorp/terraform ~> 1.2.0

cloudflare ~> 3.15.0

kubernetes ~> 2.11.0

http ~> 2.1.0

terragrunt

global/prod/terragrunt.hcl

undefined no version found

global/stag/terragrunt.hcl

undefined no version found

Check this box to trigger a request for Renovate to run again on this repository

Update 04/22/2022:

No more Vagrant with the V, now we have k3d with the D.
Update the document
Update all the values files for dev
Switch from real domain to nip.io for dev

locmai / humble Goto Github PK

humble's Introduction

Humble Project

Overview

Core Features

Documentation

Acknowledgements

Contacts

humble's People

Contributors

Stargazers

Watchers

Forkers

humble's Issues

Repository problems

Edited/Blocked

Open

Detected dependencies

Repository problems

Edited/Blocked

Ignored or Blocked

Detected dependencies

Recommend Projects

Recommend Topics

Recommend Org