Comments (11)
yoink00
commented on May 18, 2024
1
I've also faced the same problem on a setup where I need to use the non-default interface to communicate between the nodes. I've made an attempt to allow this to be configured here: https://github.com/yoink00/k3s
It is completely untested but should be complete. I'll be testing this tomorrow. Any comments would be gratefully received.
To override the default interface specify '--flannel-iface=eth1' (for example).
from k3s.
erikwilson
commented on May 18, 2024
1
Verified by setting --flannel-iface=... and checking that ip -d l show flannel.1 included the desired interface. Thanks for submitting a PR @yoink00, if this does not fill your needs feel free to re-open or submit a new issue @selcukusta & @aaliddell
from k3s.
ibuildthecloud
commented on May 18, 2024
@selcukusta There is a --node-ip x.x.x.x flag, would that work?
from k3s.
selcukusta
commented on May 18, 2024
hi @ibuildthecloud! Unfortunately, doesn't work. My cluster info:
master01 192.168.0.10
worker01 192.168.0.11
worker02 192.168.0.12
vagrant@worker01:~$ ifconfig output is;
cni0 Link encap:Ethernet HWaddr d6:40:4c:2e:17:eb
inet addr:10.42.1.1 Bcast:10.42.1.255 Mask:255.255.255.0
inet6 addr: fe80::d440:4cff:fe2e:17eb/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1450 Metric:1
RX packets:9 errors:0 dropped:0 overruns:0 frame:0
TX packets:11 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:564 (564.0 B) TX bytes:918 (918.0 B)
enp0s3 Link encap:Ethernet HWaddr 02:0d:67:c1:5e:a8
inet addr:10.0.2.15 Bcast:10.0.2.255 Mask:255.255.255.0
inet6 addr: fe80::d:67ff:fec1:5ea8/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:76692 errors:0 dropped:0 overruns:0 frame:0
TX packets:14231 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:102405443 (102.4 MB) TX bytes:1430370 (1.4 MB)
enp0s8 Link encap:Ethernet HWaddr 08:00:27:1d:57:09
inet addr:192.168.0.11 Bcast:192.168.0.255 Mask:255.255.255.0
inet6 addr: fe80::a00:27ff:fe1d:5709/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:980 errors:0 dropped:0 overruns:0 frame:0
TX packets:914 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:774461 (774.4 KB) TX bytes:152067 (152.0 KB)
flannel.1 Link encap:Ethernet HWaddr 72:60:cc:df:cb:e0
inet addr:10.42.1.0 Bcast:10.42.1.0 Mask:255.255.255.255
inet6 addr: fe80::7060:ccff:fedf:cbe0/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1450 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:8 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:1192 errors:0 dropped:0 overruns:0 frame:0
TX packets:1192 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1
RX bytes:824713 (824.7 KB) TX bytes:824713 (824.7 KB)
veth29e02690 Link encap:Ethernet HWaddr 3a:82:92:a5:3c:16
inet6 addr: fe80::3882:92ff:fea5:3c16/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1450 Metric:1
RX packets:9 errors:0 dropped:0 overruns:0 frame:0
TX packets:18 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:690 (690.0 B) TX bytes:1476 (1.4 KB)
vagrant@worker01:~$ sudo cat /run/flannel/subnet.env output is;
FLANNEL_NETWORK=10.42.0.0/16
FLANNEL_SUBNET=10.42.1.1/24
FLANNEL_MTU=1450
FLANNEL_IPMASQ=true
It might be wrong, am I right?
from k3s.
colby
commented on May 18, 2024
This is a documented behavior of kubelet, where it will select the default interface, which happens to be the NAT network of Vagrant. Try using KUBELET_EXTRA_ARGS to override that.
from k3s.
selcukusta
commented on May 18, 2024
Yes, you're right. Two actions are needed. The first one is;
sudo sed -i '9s/^/Environment="KUBELET_EXTRA_ARGS=--fail-swap-on=false --node-ip='"$1"'"\n/' /etc/systemd/system/kubelet.service.d/10-kubeadm.conf but kubelet service is embedded in k3s binary.
Another one is add --iface=enp0s8 flag to flannel manifest file like that;
containers:
- name: kube-flannel
image: quay.io/coreos/flannel:v0.10.0-amd64
command:
- /opt/bin/flanneld
args:
- --ip-masq
- --kube-subnet-mgr
- --iface=enp0s8
But the interface selection operation is embedded also. So I couldn't change them.
from k3s.
aaliddell
commented on May 18, 2024
Setting --node-ip only provides half the required configurability, as @selcukusta mentioned.
For some clarification, consider the following setup:
^ ^
| |
| |
Default route | 10.2.0.1 10.2.0.2 | Default route
10.1.0.1 | +--------------------------+ | 10.1.0.2
| | Internal Net | |
+--+-----+--+ +--+-----+--+
|eth0| |eth1| |eth1| |eth0|
+----+ +----+ +----+ +----+
| | | |
| node1 | | node2 |
| | | |
+-----------+ +-----------+
Each node has two interfaces, one to the wider network (eth0, set as default route) and one connected to a private internal network (eth1). The end goal is for the flannel VXLAN to sit on the private eth1 network, whilst the LoadBalancer services etc should sit on the eth0 interface.
To achieve this, each node should have their --node-ip set to their eth0 IP, which is what would occur by default as this is the default interface. However, the flannel VXLAN cannot be moved to the eth1 interface, as it is hardcoded to select the default route interface: https://github.com/rancher/k3s/blob/17a44a319700b69ad20ec13086e219bcceb24cb5/pkg/agent/flannel/flannel.go#L40
For evidence: setting --node-ip to 10.2.0.1 does not affect the VXLAN interface (and also obviously puts the services on the wrong ip):
ip -d l show flannel.1
4: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN mode DEFAULT group default
...
vxlan id 1 local 10.1.0.1 dev eth0 srcport 0 0 dstport 8472 ...
To solve this, somehow the flannel interface should be configurable and it looks like the config options may be available in func flannel(...) as an argument? Perhaps --flannel-iface=... as an argument?
With standard Flannel this is available as an option, which could be used as inspiration: https://github.com/coreos/flannel/blob/0fb471046932ef5f8f363f7daace3d6f4d372dc9/main.go#L118 & https://github.com/coreos/flannel/blob/0fb471046932ef5f8f363f7daace3d6f4d372dc9/main.go#L195-L216
Also, tengentially related: the readme states the VXLAN port is on 4789, but flannel puts it on 8472, as you can see in the command output above. I'll send a PR to change the docs.
from k3s.
aaliddell
commented on May 18, 2024
@yoink00: That looks reasonable, you should open a PR to get others to take a look.
In the meantime, for people stuck with this: I just went the route of using --no-flannel to bypass the fixed internal config, then manually deployed Flannel as CNI with the correct --iface option passed in. Although this also requires setting up CNI plugins correctly manually in the right places and matching the Flannel config to the CIDRs etc.
from k3s.
deniseschannon
commented on May 18, 2024
This is available in our latest RC (v0.4.0-rc3). If anyone could test it out, it'd be great!
from k3s.
balthild
commented on May 18, 2024
Does this option (--flannel-iface) can be set by environment variable?
from k3s.
costela
commented on May 18, 2024
@balthild currently no (see here), but I just opened #659 to see if we can address this.
from k3s.
Related Issues (20)
- does k3s garbage collector collects events? HOT 1
- failed to get CA certs: Get "https://127.0.0.1:6444/cacerts": read tcp HOT 1
- System upgrade controller not showing in Rancher after cluster import HOT 5
- node-restart takes over 2 minutes to shutdown due to longhoern
- Move E2E onto newer base OS
- Add E2E Split server test to PR CI HOT 1
- Errors logged when pulling images from text file HOT 1
- k3s tls certificate error HOT 2
- Auto-import images into the embedded registry HOT 4
- Use auto find runtime configfile inject in containerd config.toml then containerd startup failed HOT 2
- Services should be able to be accessed from other nodes than the one the pods are running on.
- virustotal.com flags embedded executables HOT 1
- K3s should not disable etcd and apiserver when used with kine HOT 1
- HelmChart doesn't render %{KUBERNETES_API}% value HOT 1
- Wrong Traefik Image inside static chart? HOT 2
- K3S Rootless - IPtables-restore failed HOT 6
- Agent node disk inexplicably skyrocketed and then disconnected
- etcd snapshot controller thrashes on etcdsnapshotfile management when server is run with `--disable-agent` HOT 1
- Unable to connect to node node after master upgrade
- enable kubectl debug in k3s
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from k3s.