Control planes or nodes getting unavailable over ssh about terraform-hcloud-kube-hetzner HOT 1 CLOSED

@Mammut-Felix Interesting, the only thing that comes to mind is that your base image are older and do not have semanage installed.

1. Check Image Version and Update if Necessary:

   • Determine when your current images were created. If they are a few months old, they might need an update.
   • Refer to the pinned discussion about upgrading to version 2.x for detailed steps on updating your images. Updated images shouldn't have the issue you're encountering.

2. Verify Node Status:

   • Before proceeding with any changes, ensure all nodes in your cluster are active. Use the command kubectl get nodes to check their status.

3. Backup etcd on Another Control Plane Node:

   • SSH into another control plane (cp) node, if possible.
   • Create a backup of etcd using the k3s CLI. For specific commands, consult k3s -h.

4. Increase the Count of the First cp Node Pool:

   • Adjust the node count of the first cp node pool to 2 (up from 1).
   • Apply these changes to update the cluster configuration.

5. Cordon, Drain, and Delete the Original Node:

   • Cordon the original node (the one you plan to replace) using kubectl cordon [nodename].
   • Drain the node using kubectl drain [nodename].
   • Delete the node from the cluster using kubectl delete node [nodename].
   • Remove the node from your cloud provider using hcloud delete node [nodename or identifier].

6. Recreate the Node with Terraform:

   • Run terraform apply to recreate the node.
   • This new node should initialize and automatically join the cluster, syncing with the second node in the first node pool.

7. Repeat for the Second Node in the First cp Node Pool:

   • Once the new node is up and running, repeat the cordon, drain, delete, and removal steps for the second node in the first cp node pool (the one used for initialization).

8. Restore Node Pool Count and Apply Changes:

   • After the second node is successfully replaced, reset the node count of the first cp node pool back to 1.
   • Apply these final changes to complete the process.

By following these steps, you should be able to replace the nodes effectively. Each step is crucial for maintaining cluster stability and ensuring a smooth transition to the updated nodes.

In my own testing, the method above worked. But maybe make a full cluster backup first, enhance the first step of doing an etcd backup. There is also a vmware tool called Velero that you can use to make cluster backups.

from terraform-hcloud-kube-hetzner.