RemoteMachine Setup: K0smotronControlPlane Etcd stuck restarting due to Rampant Reconciles about k0smotron HOT 4 OPEN

I think I figured out the problem. I was installing v1.0.0 of k0smotron, but the clusterctl command was installing v0.9.6 of control-plane, bootstrap, and infrastructure modules.

I rolled my install back to v0.9.6 of k0smotron and got things consistent, and now my control plane seems to be working as expected.

I still welcome any comments in case there is anything of interest in my report.

from k0smotron.

The k0s cluster has a self-signed cert and I get x509 errors as a result of that

You mean the mgmt cluster (the single node one) has self signed cert? Where did you see the x509 errors, in k0smotron controller(s)? k0smotron, as it's running in pod(s), uses normal service account access thus it should get the CA to trust automatically injected. Dunno how that could result in x509 errors 🤔

I haven't found configuration options for disabling TLS verification

You should not need to do this for the "child" controlplanes. What happens is that k0smotron/CAPI creates all the needed certs and also the kubeconfig. The kubeconfig is generated in a way where it has the needed CA in place and thus the clients should always trust it.

I was installing v1.0.0 of k0smotron, but the clusterctl command was installing v0.9.6 of control-plane, bootstrap, and infrastructure modules

hmm, not sure I read this correctly but did you essentially have two k0smotron setups installed, one with kubectl apply ... and one with clusterctl init ...?

clusterctl command was installing v0.9.6...

oh, that reminds me: we need to figure out how to automate updating of the clusterctl metadata.yaml file, too easy to forget to update it for every release 🤦

from k0smotron.

hmm, not sure I read this correctly but did you essentially have two k0smotron setups installed, one with kubectl apply ... and one with clusterctl init ...?

I think yes, this is essentially what I did. I am installing using the install.yaml, and I'm also installing additional components using clusterctl init. I end up with these four pods.

[22:41:35][root]@[worthy-aquarium-08330][~]$ kubectl -n k0smotron get pods
NAME                                                          READY   STATUS    RESTARTS   AGE
k0smotron-controller-manager-75bfdfdfdb-w6ss6                 2/2     Running   0          33h <---- created by install.yml
k0smotron-controller-manager-bootstrap-545bc97cfc-fp5p9       2/2     Running   0          33h <---- created by clusterctl bootstrap
k0smotron-controller-manager-control-plane-56669586df-2vk5c   2/2     Running   0          33h <---- created by clusterctl control-plane
k0smotron-controller-manager-infrastructure-8f8547c76-597hh   2/2     Running   0          33h <---- created by clusterctl infrastructure

I was following the guide at https://docs.k0smotron.io/stable/install/#full-installation.

I did this because when I did the "full" install via the install.yaml, I was missing CRDs that would allow me to properly set up remoteMachines. I think I was missing the clusters.cluster.x-k8s.io/v1beta1 CRD in particular. I can't remember exactly, though; I'd need to rerun my whole setup.

Ultimately, since I have a working config now, I think that I might have been creating some sort of weird version conflict between the component installed with install.yaml and the other components installed with clusterctl. It's also entirely possible that I'm doing this completely wrong and misinterpreting the installation instructions.

Kinda leads me to a few vague questions:

1. Why are CRDs "missing" from the "full" install.yaml? Is the "full" install actually full?
2. Am I improperly duplicating my install with unnecessary stuff?

from k0smotron.

Hello!

1. The k0smotron (both in full and in cluster-api installations) provides only its own CRDs. But, when you run clusterctl init the clusterctl installs also Cluster API core components and CAPI CRDs, including clusters.cluster.x-k8s.io/v1beta1
2. Potentially, it can cause some issues, since two controllers are watching the same resources. It's safe to just delete k0smotron-controller-manager deployment.

Also, we've updated the metadata.yaml file, so you can upgrade Cluster API components. I think, re-running the init command should be enough:

clusterctl init --bootstrap k0sproject-k0smotron \
                --control-plane k0sproject-k0smotron \
                --infrastructure k0sproject-k0smotron

from k0smotron.