ripcordsoftware / hvk8scluster Goto Github PK

WSL can be executed in script from the command line - so no need for a manual shell.

Invoke WSL from script to create the preseed ISO without needing user interaction at the console.

The k8s template is appox 5GB which should compress down to ~1GB. Upload the pre-build template image to GitHub and script download and extract to improve cluster build performance. Also applies to the DHCP/DNS image.

See: https://developer.github.com/v3/repos/releases/

Users should be members of the group "Hyper-V Administrators". Otherwise the PowerShell prompt will need to be elevated.

Refactor script to make the code more modular and configurable at runtime.

VM names are k8s-master, k8s-node-, etc. This is not explicit enough. Consider using hvk8s- as a prefix.

See: https://kubernetes.io/docs/setup/production-environment/container-runtimes/#docker

Based on PrepareNode.ps1 in https://github.com/kubernetes-sigs/sig-windows-tools configure the Windows node so it can be added to the cluster.

See: https://www.mirantis.com/blog/getting-started-with-mirantis-container-runtime-on-windows-server/

To include:

• number of worker nodes
• amount of node RAM
• number of node CPU cores

See: https://docs.microsoft.com/en-us/windows-server/virtualization/hyper-v/supported-debian-virtual-machines-on-hyper-v

The current script does not specify the Kubernetes version, so we end up with latest (1.20). Allow the user to specify the version they want.

The default user is hvk8s. When the cluster is deployed allow this to be specified by the user if they wish.

If a command results in an failure write the stdin/stderr to file to allow the user to troubleshoot the problem

See: https://metallb.universe.tf/installation/#installation-by-manifest

Allow ingress on common ports (80, 443, mysql, psql, mongodb, etc) with haproxy and nginx.

Requires an haproxy.conf like:

global
  daemon
  maxconn 2048

defaults
  mode tcp
  timeout connect 5000ms
  timeout client 50000ms
  timeout server 50000ms

listen http-in
  bind *:80
  server server1 172.31.0.11:80 maxconn 1024
  server server2 172.31.0.12:80 maxconn 1024
  server server3 172.31.0.13:80 maxconn 1024

... and a customised nginx ingress install like:

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm upgrade -i my-ingress-nginx --set controller.kind=DaemonSet --set controller.hostPort.enabled=true ingress-nginx/ingress-nginx

See: https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/adding-windows-nodes/
Also: https://docs.projectcalico.org/getting-started/windows-calico/

In PowerShell help command will list the parameters of a command. Add help text to scripts so they can self-describe to users.

The cluster scripts add Calico since this is required to get the cluster to schedule pods. The script applies defaults which work but may benefit from further investigation to understand their impact and possible enhancements.

Current versions:

• Rancher Wins: 0.0.4
• CNI: 0.8.7
• Flannel: 0.15.0

Consider extracting application versions from the flannel/proxy containers.

Dynamic memory seems to increase over time using all the hosts resources. Replace with fixed RAM settings since we can at least control the behaviour of the system.

The scripting does not include a storage backend. Rook with NFS exposed by the DHCP/DNS VM may allow for cluster volume creation.

Windows nodes can support Hyper-V dynamic memory, this will reduce the memory footprint when multiple Windows nodes are used.

Add a tool or command line option to capture the cluster config as part of the create process.

When Windows nodes are present Rook provisions Linux containers on those nodes.

Review the chart and example yaml usage. Consolidate the approach and fix as much as possible.

Node selection and affinity issues raised with the Rook team in: rook/rook#9165

Include reducing the provisioning memory overhead

Enabling dynamic memory (hv_balloon) with Debian 10/11 guests causes Hyper-V to continually allocate memory.

Observed behaviour is:

• Dynamic memory is enabled, min as 512MB, start as 768MB and max as 8192MB
• After booting htop (etc.) shows the memory is limited to the start size
• Opening applications like Firefox, Libre Office and GIMP will result in an OOM and an apparent desktop logout
• Meanwhile memory allocation in Hyper-V and Windows is increasing up to the max limit, however the VM never sees the RAM and will always OOM when opening challenging apps
• Increasing the max to 32GB shows the RAM usage increase continually even when there is no desktop application activity, again the max limit of 32GB can be hit without the VM seeing the RAM and OOMing when apps are opened

Detect the k8s-template if already present and use it by default. Allow the user to override template cloning and build each node from the preseed.

Useful start point: https://gist.github.com/WillemRB/5eb18301462ed6eb23bf

Releasing the DHCP lease via ssh on Windows is problematic. Add a service to perform the release and restart which can be manually started from ssh allowing the client to drop the connection before the reboot happens.