DNS failures for internal AWS URLs about dns HOT 16 CLOSED

One more point of context: I'm running pods (instances of the Datadog agent) which are doing lookups of these URLs regularly, several times a minute.

from dns.

from dns.

Yep, these all resolve correctly (and consistently) through the servers specified in /etc/resolv.conf.

Incidentally, kube-dns doesn't seem to return the 'authority' section of its responses consistently, either. I'm not sure what that means, or if it's a design choice.

Edited: /etc/resolv.conf on the EC2 instances. /etc/resolv.conf in the running containers only contains the kube-dns IP, as expected.

from dns.

We're also seeing this now for some other internal DNS entries we've created ourselves - it's not just the AWS-created ones.

from dns.

One more datapoint, if it's helpful: I've occasionally seen responses via dig that show an 'authority' section but no 'answer' section. This seems especially weird.

from dns.

I've convinced myself the problem is how the flags being passed to dnsmasq interoperate with private DNS servers, not with the core binaries in kube-dns.

I'm running a test configuration right now that's showing some promise. I'll close this bug if it continues without error overnight, and follow up with the kops people.

Thanks for taking a look!

from dns.

Was there a recent upgrade to 1.6? That is the first version where dnsmasq is configured differently.

from dns.

We're running a brand-new cluster, and ran into this issue when we starting running real workloads on it. It's always been 1.6.

from dns.

Are you using the new stub domains feature?

from dns.

I don't believe so. We're using kops to do setup - see the template file I linked at the top of the bug.

My current theory is that my bug is caused by our /etc/resolv.conf, which looks like:

nameserver 10.16.0.4
nameserver 8.8.8.8
search internal-name us-west-2.compute.internal

dnsmasq will use the configured nameservers to do lookups - but the problem is that it will choose the server arbitrarily / randomly, and caches negative lookups. If it picks Google's DNS server (8.8.8.8) for resolving internal URLs, it will get a negative result, and return & save that entry.

My current solution is to add the -o flag to the dnsmasq startup flags, which tells it to use the DNS servers in-order. Our internal nameserver will normally resolve all DNS entries fine, so this is the configuration we want. I think this is a much more sensible default than without the flag.

from dns.

It might be worthwhile to take a look at http://blog.kubernetes.io/2017/04/configuring-private-dns-zones-upstream-nameservers-kubernetes.html.

You can send all requests for *.compute.internal to 10.16.0.4 and keep 8.8.8.8 as the upstream nameserver.

from dns.

Awesome, thank you - that's extremely helpful.

from dns.

I'm closing this in favor of the new kops bug.

from dns.

So this isn't the default AWS configuration, AFAICT, in that this will only happen if you explicitly add 8.8.8.8 to the AWS DHCP options. But it sounds like that is safe with "normal" DNS, but not safe with dnsmasq, because of the randomization?

Is that right @bowei ? If so we can indeed add workarounds using stub domains, but this seems like it will hit a lot of people in the real world - and not just with AWS. And you can have private Route53 domains on AWS, so it isn't a simple finite set like *.compute.internal

from dns.

It's never a good idea to mix split horizon name servers in your resolution path. Some libc query sequentially, round robin, musl sends queries in parallel (!). It can result in unpredictable behavior.

from dns.

It might be worthwhile to take a look at https://blog.kubernetes.io/2017/04/configuring-private-dns-zones-upstream-nameservers-kubernetes.html.

You can send all requests for *.compute.internal to 10.16.0.4 and keep 8.8.8.8 as the upstream nameserver.

This link is 404 now.
Below is the updated url:
https://kubernetes.io/blog/2017/04/configuring-private-dns-zones-upstream-nameservers-kubernetes/

from dns.