Failed on 'first_control_plane` about terraform-hcloud-kube-hetzner HOT 19 CLOSED

yes sirr!! That was the cause..

thank you so much for the pointer! 🙏
will continue playing around with your repo in the meantime ;)

from terraform-hcloud-kube-hetzner.

thank you @mysticaltech !

I was planning to use Fedora for the production environment due to the higher release on the Linux kernel as you mentioned.
However, I did a bit of investigation and many said that Fedora could be a little unstable as it is always the latest and the greatest which may also introduce bugs.

Fedora also forces upgrade every year which will take down the node for sometime and unnoticed - for production environment, yikesss!!!

this time around is my first time using Fedora - are you aware of such an issue?
I was thinking to keep the Fedora for my development environment until I have a better idea on the stability of it.

But would love to hear your thoughts on it!

from terraform-hcloud-kube-hetzner.

yes, that was my point of concern as well.

With that in mind, I was thinking to experiment with Fedora in the development environment and see how severe this upgrade process is.
Many mentioned that they do it at least once a year (could be twice if they found a major bug or security issue).

I will run the same testing, probably will have to wait for 12 months prior making a decision to move the production environment to Fedora.
Seems like you're a head with the dev environment already with 6 months in the Fedora 34.

I'll keep you posted on my findings - and hope you do too! :)

from terraform-hcloud-kube-hetzner.

Yes, definitely will do! About security upgrades for both systems, it's best to set up automatic upgrades every day as to not leave the system vulnerable, and that's what I do for Fedora in this repo, otherwise, for Ubuntu, see how to install and configure unattended-upgrades.

from terraform-hcloud-kube-hetzner.

@kiwinesian Just so you know, just fixed a bunch of issues. Actually, the internal k3s process was communicating with their external IPs! 🤦🏻 Found that out when I added and activated the Hetzner firewall to the config. Since I turn off the firewall at the OS layer, I set it up from Hetzner and assigned it to all servers, then all kinds of issues showed up. Turned out I was not configuring k3s well. Now all fixed and a lot more robust, accurate and secure!

from terraform-hcloud-kube-hetzner.

ha! i was just about to get into setting up firewall, but did notice that the kubelet was not explicitly declared in the init-config.

Will do a pull ;)

from terraform-hcloud-kube-hetzner.

Ah, you mean the kubelet-arg? Check this line

from terraform-hcloud-kube-hetzner.

@kiwinesian I am happy to report that after the latest adjustments, Cilium's native routing now works out of the box, so I have disabled the tunnel.

Also switched to DSR for the Kube-proxy internal load balancer (don't ask me details, I just know it's better than the previously used SNAT 😂, but required native routing to be activated first).

from terraform-hcloud-kube-hetzner.

oh thank youuu for keeping me updated!

I haven't even had a chance to do a pull post the firewall changes. ;P
Hummm i haven't look into the DSR details - but thought kube-proxy was disabled and therefor was not running SNAT?

aahh yess, i think we declared --node-ip=x.x.x.x to ensure that the node is running with internal IP for the k3s communication.
I wasn't aware of any approach on how to set it otherwise.

EDIT: ohh, I just saw how you set --node-ip on each .tf file

Ah, you mean the kubelet-arg? Check this line

from terraform-hcloud-kube-hetzner.

heyy,,

I cloned your repo and just tried as it is - can confirm that the csi is working!
hmmm.. maybe something on my code base 🤔

on a side note, would you deploy k3s over k8s in production?
i think k3s doesn't use etcd which may hinder optimum performance of the control plane.

let me know what your thoughts are! :D

from terraform-hcloud-kube-hetzner.

heyy buddd @mysticaltech

so sorry for going AWOL to your notes - very kind of you for keeping me updated your project!
Was a little busy with work during that time and then took a little time off.

Only to remember that I haven't replied to your message!

i have never seen k3os before - might check it out!
Is it any better compared to regular k3s?

I kinda love Cillium actually.. hehe ;p
ill take the hit on the latency just to have Cillium.

btw, what's your thoughts on running k3s instead of k8s for full production load?
this will be a heavy load, potentially up to 200 nodes per cluster. will definitely need to be multi master instead of a single master

great stuff bud!

from terraform-hcloud-kube-hetzner.

@kiwinesian That is probably caused by the reboot in the previous block. It works well with Fedora, but you may want to remove the reboot on Ubuntu, especially if you do not upgrade, it may not be needed. Try removing it! Look for the "shutdown" command.

from terraform-hcloud-kube-hetzner.

Awesome, note that to take advantage of the full goodness of Cilium like BPF, it is important to use a Linux kernel > 5.10, which is not the case for current versions of Ubuntu.

On the other hand, Fedora is the distro used by Linus Torvalds himself and supported by RedHat, so always up to date with the latest and greatest!

from terraform-hcloud-kube-hetzner.

@kiwinesian Very interesting point, IMHO, for something as fundamental as running containers and Kube, I think it should remain near 100% stable. I personally never used it in production either, but I intend to do so.

Probably desktop usage is a lot more complicated than was we are trying to use here. But even with that, I have been running pretty stable for the last 6 months on my personal machine. I think it's pretty safe to bet that Fedora server stable releases as given by Hetzner and should be really ok.

from terraform-hcloud-kube-hetzner.

sooo got time to rebase my code, but looks like Cilium is not quite happy.

I'm not sure whether this is because the tunnel being disabled + native-routing.
this was continuous challenge in getting native-routing running for Cilium :(

did you run into such an issue?

from terraform-hcloud-kube-hetzner.

Ah, my pleasure! Glad to see that you've given Fedora a chance hehe! For the csi crash, I haven't had such issues. Make sure that the firewall is set up well and opens the required ports for it to function well, see https://github.com/mysticaltech/kube-hetzner/blob/6faccbc721941687757afb405a8036eb3927117b/main.tf#L23-L80

from terraform-hcloud-kube-hetzner.

Ah very happy to hear that it is now working @kiwinesian! So actually no this is pure k3s, it turns out that they moved away from their previous default dqlite and now use "embedded etcd", so we do not really touch it. It was just something that came out in the logs when I was trying to set up the firewall.

from terraform-hcloud-kube-hetzner.

Hey @kiwinesian, this kinda has become our messaging thread haha! Just like to keep you updated as one of the first users of this project. I completely revamped the system. Switched to k3os for the underlying nodes, removed cilium, while ensuring that k3s still use the Hetzner private network for low latency internode communication without the need for encryption.

This has the benefit of implying everything, as no node maintenance is needed, both k3os and k3s get upgraded automatically in a fully HA fashion! This should now be a just works experience, at least it was the aim! Please test if you feel like it and let me know.

from terraform-hcloud-kube-hetzner.

Hey @kiwinesian, no problem at all! Good to hear back from you.

k3os is good, as is it a container Linux that integrates very well with k3s (so upgrades are just seamless and automatic both for the node and k3s). The only thing, is Rancher was recently bought by SUSE, and they've dropped official support for k3os 🤦🏻, so right now updates are rare and done by the maintainer on the weekend. So will have to find another one like Fedora CoreOS maybe.

Glad to see you are still running Cilium, it's actually great! Just wanted to simplify things, but the good thing is that the work we did on Cilium thought me how to use the Hetzner underlying private network correctly. That was a huge deal!

About k8s, the only thing I know is that k3s merges code from k8s directly, on a regular basis. So it's probably they share a lot of the code base, it's just that k3s removes a few old artifacts and compiles everything onto the same binary. So it's IMHO it has good, if not faster and lighter.

from terraform-hcloud-kube-hetzner.