Can't establish direct connection to a Kubernetes pod despite having a direct connection to its node (Flannel CNI) about tailscale HOT 9 OPEN

how to properly quantify how hard its NAT is

If you take a look at your client in Tailscale Machines panel, there is a field named varies, if that's set to true, you're behind what we consider hard nat.
However, looking at your debug logs it actually doesn't seem to be the case

from tailscale.

Small update: I have switched to Calico for networking on my cluster, since it has a way to disable source port randomization, by configuring Felix (Calico's node agent) to override iptables feature detection and make it think the randomization is not supported.

The relevant bits of documentation can be found here: https://docs.tigera.io/calico/latest/reference/felix/configuration. The relevant config key is referenced by some variation of featureDetectOverride, based on the method it's configured (env vars, config file, the Calico k8s operator, etc.), and the specific feature flag is MASQFullyRandom.

With this set, I can now directly connect to my pods, and iptables only reports a single rule with --random-fully, presumably added by Kubernetes itself:

root@control01:~# iptables-save | grep random-fully
-A KUBE-POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -j MASQUERADE --random-fully

I'm guessing if someone removed the rules created by Flannel while also preventing it from adding them back, it would have the same effect.

Should I close this issue now? My issue is fixed but I suppose it could also track any changes to the docs.

from tailscale.

Hi @LiquidPL thank you very much for the confirmation that you now get direct connections with Calico.

Should I close this issue now? My issue is fixed but I suppose it could also track any changes to the docs.

I think we could leave this open- as you say it would be great to update docs and I would also like to reach out to Flannel folks and see if they might be willing to make the port randomization for external connections configurable.

from tailscale.

Hi, thanks for creating the issue.

A known issue why it is not possible to get direct connections on some Kubernetes configurations is when the CNI enforces source port randomization.
I am not very familiar with Flannel, but it appears that they added support for randomization a while ago flannel-io/flannel#1004

You could verify that it is on, by grepping for random-fully in your iptables rules on nodes - if there is a SNAT rule that would apply to traffic originating at Pods going out to the internet with `random-fully that would likely be a reason why tailscale running in that Pod cannot get direct connections.

We do need to document this behaviour and/or if it can be turned off for different CNIs.
It might be possible to turn this off for Flannel.

See also #3822

from tailscale.

Yeah, I have found rules like that in my iptables:

root@control01:~# iptables-save | grep random-fully
-A FLANNEL-POSTRTG -s 10.42.0.0/16 ! -d 224.0.0.0/4 -m comment --comment "flanneld masq" -j MASQUERADE --random-fully
-A FLANNEL-POSTRTG ! -s 10.42.0.0/16 -d 10.42.0.0/16 -m comment --comment "flanneld masq" -j MASQUERADE --random-fully
-A KUBE-POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -j MASQUERADE --random-fully

I'll try removing the --random-fully from these rules and see what happens.

Additionally, I've looked into how Flannel builds its iptables rules, and it appears that it will automatically enable port randomization if it finds that iptables supports it. So I'm not sure if there is a way to easily disable it, and I'm also not sure if it's a good thing to completely disable it either (although I am not a network person so I dunno).

from tailscale.

Additionally, I've looked into how Flannel builds its iptables rules, and it appears that it will automatically enable port randomization if it finds that iptables supports it. So I'm not sure if there is a way to easily disable it, and I'm also not sure if it's a good thing to completely disable it either (although I am not a network person so I dunno).

Thank you for taking a look- it does indeed seem like they have it hardcoded.

Is the client that you are connecting from also behind hard nat?

from tailscale.

Yeah, most likely - it's a student dorm network, and while I have no idea how to properly quantify how hard its NAT is, I remember I've always had issues connecting to P2P games on a Nintendo Switch, so there's certainly some stuff going on there.

from tailscale.

I have tried manually editing the relevant iptables rules but either I'm doing something wrong and these are not applied, or flannel just keeps reapplying them, because I am still unable to get a direct connection. I guess I will have to manually patch flannel itself and try again.

Though I suppose this should be reported upstream as well.

from tailscale.

Hi @LiquidPL thank you very much for the confirmation that you now get direct connections with Calico.

Should I close this issue now? My issue is fixed but I suppose it could also track any changes to the docs.

I think we could leave this open- as you say it would be great to update docs and I would also like to reach out to Flannel folks and see if they might be willing to make the port randomization for external connections configurable.

Did you reach out to Flannel folks about this? I am definitely interested as well!

from tailscale.