Self-hosted kubelet breaks port-forwarding about bootkube HOT 16 CLOSED

Potentially related: kubernetes/kubernetes#19765

from bootkube.

see kubernetes/kubernetes#26093

from bootkube.

this is a tricky one. as kubernetes/kubernetes#26093 clearly shows over time kubelet gained a fair share of external dependencies, some of which may come with a non trivial tail. Given the focus on slimness and lightness not all packages that own those external dependencies ship with CoreOS (socat is the issue here).

on production this is easily solved by going the hyperkube way (as those dependencies stop being an CoreOS concern as they just become containerized) but here we have an issue because we are running things directly on top of CoreOS.

So, IMHO, one of two routes would have to be paved - either one assumes that all (that matter to us) dependencies of k8s should ship with stock CoreOS (fattens the OS a bit, and is a non backward compatible trick) and the issue is over or change the tooling so that a containerized bootkube is run so that when it runs there would be worries of any kind about what userland is or isn't available.

from bootkube.

I'm fairly certain socat is in the hyperkube image, and we're running both kubelets via the hyperkube image. So I'm fairly certain it isn't a problem of the image, but rather how the self hosted-kubelet is configured/ran. I'm guessing it has to do with the nsenter stuff that's done as the entrypoint to hyperkube, but I'm not certain.

from bootkube.

So, the nsenter prefix is messing with PATH resolution. Specifically, we lose the binaries installed into the container filesystem when we break out of the container mount namespace.

We could:

1. Attempt to use the --containerized flag, which will prefix mount commands with nsenter so they take effect on the host namespace (note: this is marked as experimental).
2. Still use global nsenter but chroot to the container working directory (e.g., function in host mount namespace, but use container filesystem as root).
3. (End goal, eventually) run kubelet via rkt fly (similar to host).

from bootkube.

To reiterate the problems:

1. We want to be on the host mount namespace so that any mounts we make are visible to docker. K8s doesn't support mount propogation so mounts must be made from the host mount namespace (or kubernetes must support shared mounts)
2. We package socat in our container's filesystem, but it's not on the host's filesystem. Being in the host mount namespace results in socat not being in our path
   2a) If we do chroot into our container rootfs (socat is now in our path woo) but remain in the host mount namespace to solve 1), we no longer have the bindmounts for kubelet (e.g. /etc/kubernetes) that are in the container's mount namespace since chrooting doesn't get them back.
   2b) We could re-create all bindmounts we need + chroot. Congrats, we just reimplemented rkt fly, let's just use nsenter + kubelet-wrapper in that case... which we could package in the image I suppose

One other possible solution would be to explicitly nsenter, not chroot, and add our PATH (/container/usr/bin ...) Unfortunately, this is fragile because you're still using the host's lib directory. You could also modify the LD_LIBRARY_PATH but I feel like this is a path to bad ideas, if you'll pardon the pun.

I'd like to disagree that the end goal is kubelet in rkt fly here. I think the end goal should be kubelet, but smart enough to run in a container appropriately. The --containerized nsenterMounter is in that direction, and so if that works I think it's an okay step. The downside with that approach is it prevents us from packaging additional mount. commands in the container, which kidna defeats the distribution bit. Perhaps a better solution would be instead of an nsenterMounter hack, a execWithPath hack that effectively does PATH=$containerPath LD_LIBRARY_PATH=$containerLibPath /usr/bin/nsenter -t 1 -m -- $cmd

This is really a bunch of words to say that I like the direction of --containerized

from bootkube.

FYI I got this to sorta work using the --containerized flag approach with some questionable log lines.

Gist of the kubelet manifest + logs:
https://gist.github.com/chancez/ea9002c9ff6b546fe253f49165185210

from bootkube.

I made a PR for it yesterday and uploaded it now, it should work (hadn't got time to test)

from bootkube.

So to sum up our options right now:

1. use the --containerized flag and stop entering host mount namespace

Mostly depends on how stable an option we consider --containerized. It's been a while since I've looked at this flag -- so would need to reassess any known issues.

2. Continue entering host mount namespace, but change PATH to the container filesystem.

Downside here is that all binaries will use host shared libraries (increases risk of a CoreOS update breaking a previously working kubernetes install)

3. Wait on mount propagation to make it upstream

This could be a while

4. Get weird with LD_LIBRARY_PATH hacks:

bind mount a socat-wrapper script to /usr/bin/socat in container fs, and update PATH to include this explicitly.

LD_LIBRARY_PATH=./lib/x86_64-linux-gnu/:./usr/lib/x86_64-linux-gnu/ ./lib/x86_64-linux-gnu/ld-2.19.so --inhibit-cache ./usr/bin/socat

Then we would also need to do the same for other kubelet dependencies (e.g. mount tools).

Thoughts @luxas @chancez @euank ?

from bootkube.

These all make me want to cry 😿

from bootkube.

I think 1) is the least awful, but then we're relying on the host to provide mount tooling since --containerized results in Kubelet nsentering the host to run mount. Not sure if we're okay with this or what.

from bootkube.

For nfs it would work, for others tools that we don't ship in OS it would be a problem...

from bootkube.

Another option is to test nsenter --target=1 --mount --wd=. -- chroot . /kubelet $@

from bootkube.

That might be best, i think it would need some serious testing to see how that interacts with the various libraries/binaries.

from bootkube.

The natural extension of the nsenter .. chroot one is the nsenter -t 1 --mount --wd=. rkt run --stage1=fly, which naturally extends to nsenter ... kubelet-wrapper.

2. Continue entering host mount namespace, but change PATH to the container filesystem.

We know this breaks socat today, right? That sounds super fragile and the compatibility matrix would be a nightmare.

3. Wait on mount propagation to make it upstream

There might be a way to break out and fix the mount propogation to be shared while we're suggesting other horribly hacky fixes, though I can't claim to know off the top of my head how to.

from bootkube.

Closing, as this issue would be resolved by the switch to the --containerized flag implemented in #247

There may be other issues with the --containerized flag, but those should have their own issue (port forwarding issue due to socat path should no longer be an issue

from bootkube.