Comments (5)
Any update on this? I'm facing exactly the same problem
Tested on v0.2.1 and v0.2.0
from k3os.
OK, found it.
TL;DR There's probably something up with your DNS settings, which you're probably getting via DHCP. Test this by changing /etc/resolv.conf
to just read nameserver 8.8.8.8
and removing any search
statements and see if everything starts working within a couple of mins (ie kubectl get nodes
works).
Issue
The containerd configuration file in /var/lib/rancher/k3s/agent/etc/containerd/config.toml
refers to a listen address as the unqualified hostname of the k3os machine.
So containerd is trying to resolve this name into an IP, so that it knows what IP it needs to listen on for starting the service.
The issue occurs if you have a search path configured in your DNS settings (or the ones delivered by your DHCP server), AND the domain in your search path also happens to respond with an IP, for example if the search path you have configured has a wildcard DNS record, so anything.search.path.domain.com will respond with an answer.
This is what was happening to me, i get a cloudflare IP back because of a wildcard DNS entry i configured years ago, and so containerd then tries to use it as the IP to listen on, which obviously fails because my k3os machine doesn't have a local IP matching that.
What should happen, is nothing should respond, the DNS query will fall through to the local hosts file, without a domain, where it will match localhost, resolve to 127.0.0.1
and containerd will start listening properly.
Although why that couldn't just be hard-coded to 127.0.0.1
i dont know
Hope this helps someone else! I really should clean up my DNS lmao.
from k3os.
I haven't played with it since. I've got some new bare metal servers coming in next week, and I'm thinking I'll try running this on KVM, see if the issue still persists.
from k3os.
+1 same issue. Both live booted and once installed. Occurs on KVM-based and bare metal.
It seems containerd is crashing on startup as it's trying to listen on a very random IP which the host doesn't own.
This comes from the log in /var/lib/rancher/k3s/agent/containerd/containerd.log
.
Adding this IP as a loopback (so that it can be used to open a listening tcp socket) works around the issue, the node comes up and kubectl can be used;
k3os-16394 [/var/lib/rancher/k3s/agent/containerd]# ip addr add 104.18.46.239/32 dev lo
k3os-16394 [/var/lib/rancher/k3s/agent/containerd]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
k3os-16394 Ready <none> 6m27s v1.14.1-k3s.4
Confirmed this in both the live and installed versions of the 2.0 release.
Still hunting down where the IP is coming from, i haven't found a containerd config file as such yet.
containerd appears to be started via the k3s server
command which is called by this init script:
cat /etc/init.d/k3s-service
#!/sbin/openrc-run
depend() {
after net-online
need net
}
start_pre() {
rm -f /tmp/k3s.*
}
supervisor=supervise-daemon
name="k3s-service"
command="/sbin/k3s"
command_args="server >>/var/log/k3s-service.log 2>&1"
pidfile="/var/run/k3s-service.pid"
respawn_delay=5
--SNIP--
IP belongs to cloudflare, wondering if it's an "are we online" kinda check in the startup scripts which then gets wrongly used in a config template. I'll keep digging.
from k3os.
Fixed via #170
from k3os.
Related Issues (20)
- Target platform
- Token not created on install, only some config options picked up on install
- How to add tls-san to a running k3os HOT 1
- ETCD role now showing in a bare metal K3os install
- How to mount VMWare shared folder?
- The project is dead but ready to get into contributor mode HOT 47
- Configure Private Registry without TLS HOT 1
- macvlan support HOT 2
- Unable to restart connman HOT 1
- Network is not ready when executing run_cmd commands
- Is k3os still being maintained? HOT 1
- Network between pods (different nodes) is unreachable
- set fake ntp while no internet access and no NTP access
- installation issues regarding missing `growpart`
- K3os installer/deployer for Hypervisors
- Depreciated this repo / Mark as depreciated HOT 1
- bug: Previous pods still exist after a reinstallation HOT 1
- openvswitch
- openvswitch
- How to Install AppArmor on k3os
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from k3os.