Comments (11)
I tried it out on Ubuntu 16.04.6 LTS with v0.3.0 (9a1a1ec)
since the final 0.3 got released a few hours ago. Using curl -sfL https://get.k3s.io | K3S_RESOLV_CONFIG=192.168.0.19 sh -
and removing my sed workaround from cm/coredns
it works, but only without providing a custom TLD:
root@rocket-chat:/# ping my-pc
PING my-pc.fritz.box (192.168.0.20) 56(84) bytes of data.
64 bytes from my-PC.fritz.box (192.168.0.20): icmp_seq=1 ttl=61 time=0.787 ms
But when I try ping my-pc.fritz.box
it can't resolve. nslookup
also timed out:
root@rocket-chat:/# nslookup my-pc.fritz.box
;; connection timed out; no servers could be reached
Using other machines in the same networks that use 192.168.0.19
as dns-server, both domains were resolved successfully. Altough inside Vagrant I'm able to resolve my-pc.fritz.box
, it may has something to do that I'm trying this in Vagrant on Ubuntu 18.04. Content of /etc/resolv.conf
inside vagrant:
nameserver 10.0.2.2
search fritz.box
Update: It's a Kubernetes issue
Found out that this was caused by Kubernetes ndots config. Per default, we have options ndots:5
set in resolv.conf. This means that dns names must contain at least five dots before they were processed as an absolute name. my-pc
doesn't contain any dots, so it's resolved absolute by our upsteam 192.168.0.19
where we have an alias without .fritz.box
suffix by default.
But my-pc.fritz.box
contains two dots. The default setting is ndots:1
so that any dns name with at least one dot would be resolved as absolute dns. Since Kubernetes has ndots:5
the my-pc.fritz.box
is resolved as relative dns. So it would apply all suffixes from search
. This can't work since it would apply another .fritz.box
suffix, so my-pc.fritz.box
would become my-pc.fritz.box.fritz.box
.
I assume that this should speed up things for internal cluster dns entrys. But for external dns, it can slow down things. Using apt-get
for installing some debug packages like netutils
was very slow. Since I switched to the default ndots:5
it got pretty fast like on my working machine. You can also find blog posts about this issue. But in my case, the primary problem was that it breaks my absolute external dns entrys.
To solve this, customize the pods dns configuration by applying it to container
level at the pods definition:
containers:
# ...
dnsConfig:
options:
- name: ndots
value: "1"
But regarding to Kubernetes own dns, I'd consider this as a workaround for local purpose since I'm not completely aware of the productive peformance yet. As another solution, we may force absolute domain names by a leading dot.
Currently, I'm using thednsConfig
entry and dns works well with my custom server. So this problem wasn't related to k3s directly and the fix in 0.3 works well :)
from k3s.
Not completely on topic, but the fact that issue 53 is related to DNS issues sounds done on purpose :)
from k3s.
@latchmihay We may have hard coded 1.1.1.1. We will make that configurable. The default behavior of k8s is to use the hosts /etc/resolv.conf as the upstream DNS but because of systemd-resolved being the default these days (and older dnsmasq setups) it is typically 127.0.0.x IP and then breaks. So it's super hard in general to figure out what the upstream DNS should actually be. So we probably hardcoded it to 1.1.1.1.
We will add this as an option to the agent and also document it.
from k3s.
i fixed it changing the configmap for coredns from 1.1.1.1 to 8.8.8.8 ... for whatever reason 1.1.1.1:53
I could not reach
from k3s.
We have created a release candidate v0.3.0-rc3 which will hopefully fix these DNS issues. Please try it out and let me know if it helps!
The settings are configurable in that we will either take a --resolv-conf flag to pass down to the kubelet, or a K3S_RESOLV_CONF environment variable will work also. We now try to use system resolv.conf files (from /etc & systemd), and will create a /tmp/k3s-resolv.conf file with nameserver 8.8.8.8 if nameservers in the system files are not global unicast ips.
from k3s.
Is there any way to set the dnsConfig
options globally instead of on a per-pod basis?
from k3s.
For anyone arriving here from a search engine, I was able to resolve my cluster's DNS issues by
(a) using the legacy iptables rather than nftables, (b) ensuring the CNI is correctly installed (I use Calico with hardware that has multiple NICs and this requires additional setup for IP detection), and (c) flushing the iptables leftover from the CNI in between cluster installs.
iptables --version
# iptables v1.8.7 (legacy)
iptables -P INPUT ACCEPT
iptables -P FORWARD ACCEPT
iptables -P OUTPUT ACCEPT
iptables -t nat -F
iptables -t mangle -F
iptables -F
iptables -X
# ... Install k3s
from k3s.
I am not sure what 1.1.1.1 is and where its coming from
This is the CloudFlare DNS public service, like the Google DNS 8.8.8.8
from k3s.
Hmm, its probably being blocked on my network. Any idea how its being configured and how I could change it?
from k3s.
@ibuildthecloud Please maintain the current way, make configurable, but if you keep save a log of time because of the issue that you said, I've got it many time, and need be a new step on the new servers installations...
Anyway thank you, I was hoping it to migrate to the new Rancher v2.
from k3s.
This can be done by replacing proxy . 1.1.1.1
with your own dns server in cm coredns
. I wrote a detailled guide how to change this manually and automated for tools like ansible here: https://devops.stackexchange.com/a/6521/6923
from k3s.
Related Issues (20)
- Use private ip with external cloud manager that doesn't know about that ip
- Allow setting `priorityClassName` on ServiceLB daemonset. HOT 5
- k3s cluster should use GPU without mentioning runtimeClassName: "nvidia" in pod specs HOT 3
- Fail to run static pod without master running HOT 2
- --node-external-ip can not add annotate flannel.alpha.coreos.com/public-ip-overwrite HOT 1
- Pods are going into pending state after upgrading from v1.26.12-k3s1 to v1.27.11-k3s1 and v1.28.5-k3s1 (Issue is quite random)
- leader-elected etcd controllers not consistently functional when leader election/lease mismatches occur HOT 4
- Update minio-go to `v7.0.70` or newer HOT 2
- Embedded helm controller does not track owner references properly
- Network policy in dual-stack cluster stops allowing ingress after pod restart HOT 14
- ingresses stop working every now and then HOT 8
- Create a new single node cluster with embedded registry mirror does not work HOT 5
- K3S startup stuck in a deadlock when a KMS provider is configured and the node is rebooted HOT 4
- Expose k3s metrics over http HOT 2
- Capturing K3s metrics over http/https from VM HOT 1
- Agent node IP not replaced with Tailscale VPN IP HOT 7
- Pods not always creating logs HOT 1
- Best practice of setting the cluster name at deployment level HOT 4
- k3s crashes if I do not start the cloud-provider
- sh: 0: cannot open –: No such file
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from k3s.