Erroneously filtering out non-local IPv6 DNS servers when using user-defined networks about moby HOT 7 OPEN

Hi @maleadt - thank you for raising this ...

As a (rather surprising) workaround, please could you try adding --dns-search=. to your docker run command.

That should result in the container's resolv.conf only listing Docker's internal nameserver 127.0.0.11. That internal resolver will use systemd's 127.0.0.53, which I think is what you need?

Please let me know how you get on ... I'm planning to look at improving the way DNS is configured in the new year.

from moby.

Great - thank you for the update, I'm glad the workaround helps for now!

Using the Google nameservers in this case is a bug that we'll fix. But DNS config is a property of the container rather than the network ... I think it has to be, because the container can connect to more than one network, and the OS in the container can only be set up one way.

from moby.

For my own reference/memory ...

With systemd's resolver set up:

• /etc/resolv.conf is link to /run/systemd/resolve/stub-resolv.conf.
• /run/systemd/resolve/stub-resolv.conf lists only nameserver 127.0.0.53.
• /run/systemd/resolve/resolv.conf lists the nameservers systemd will use (192.168.0.2 and the LL address).

For the default network:

daemon/container_operations.go:setupPathsAndSandboxOptions() calls libnetwork.OptionOriginResolvConfPath(cfg.GetResolvConf()).

GetResolvConf() returns a path that's been set up by resolvconf.Path(), which had figured out that systemd is running. So it selected alternatePath="/run/systemd/resolve/resolv.conf" instead of /etc/resolv.conf.

Because it's for the default network, legacy networking, that file is just copied - there's no attempt to insert the internal resolver. So, the container ends up with a resolv.conf containing the nameservers that'll be used by systemd.

For a user-defined network:

setupPathsAndSandboxOptions() calls libnetwork.OptionOriginResolvConfPath("/etc/resolv.conf").

Then, libnetwork's Sandbox.setupDNS() calls resolvconf.FilterResolvDNS(). It's following systemd's link to stub-resolv.conf, so it only sees systemd's nameserver 127.0.0.53.

FilterResolvDNS() removes that localhost address, because it won't be visible from the container's network namespace. So, it ends up with an empty list of nameservers and "fixes" that by adding Google's nameservers... IPv6 and IPv4. (It could have just added our internal resolver?)

Later, incidentally ...
Daemon.connectToNetwork()
-> Endpoint.Join()
-> Endpoint.sbJoin()
-> Sandbox.updateDNS()
-> resolvconf.FilterResolvDNS()

That second call to FilterResolvDNS() doesn't change the container's resolv.conf, it's harmless, but I don't think it needed to be called a second time. (Perhaps it was meant to be looking for addition of a first IPv6 endpoint to an existing sandbox... can be removed if I end up keeping the IPv6 nameservers with the IPv4 ones, in the internal resolver.)

Then, during the container-start, Sandbox.StartResolver() calls Sandbox.rebuildDNS(), which replaces the IPv4 nameservers with the internal resolver's 127.0.0.11 - it doesn't touch the IPv6 nameservers. The IPv4 Google nameservers are discarded. The internal resolver uses the nameserver found by Sandbox.setupDNS(), systemd's in this case.

The proposed workaround:

Setting dns-search, causes Sandbox.setupDNS() not to try to filter the host's resolv.conf (no call to resolvconf.FilterResolvDNS()).

Instead it just adds servers from the host's /etc/resolv.conf which, as above, is the link to systemd's stub_resolv.conf. So, it adds 127.0.0.53.

Then, the call to Sandbox.rebuildDNS() during container-start sorts things out - in this case it replaces 127.0.0.53, making that the internal resolver's upstream server.

(Shudder.)

from moby.

Ah! Nice finds.

Yes, it looks like logic that was intended for the default bridge and other networks got intertwined.

For the default bridge there's no internal DNS (currently!) so we configure the container with the host's resolvers (excluding localhost because that can't work)

For other networks, there's the internal DNS, which forwards requests to the host's resolver.

It looks like we're doing way too much in the latter case; effectively all we'd need is the internal DNS, nothing more (but maybe it needs to listen on an IPv6 loop back as well?).

Or are there other options (DNS search) that must be inherited from the host and configured in the container? (I'd somewhat expect that to be opaque to the container, at most handled by the embedded DNS)

from moby.

Or are there other options (DNS search) that must be inherited from the host and configured in the container? (I'd somewhat expect that to be opaque to the container, at most handled by the embedded DNS)

My 2 cents ...

We leave the 'search' up to the resolver function in the container at the moment ... whether we continue to do that needs careful thought.

A resolver function like libc's or Go's is doing a lookup for a plain name (not a DNS request). It may do things like looking at 'nsswitch.conf', returning results from the host's '/etc/hosts', and running through 'search' domains.

But the engine's internal resolver gets a DNS request. Generating a DNS response with an answer we found in a different domain would be a bit odd/wrong. But, it'd probably deliver the end result the application in the container wants, nearly all of the time.

In rootless mode, the internal DNS will currently respond with 'hosts' entries (like DD). Also, the internal resolver has its equivalent of built-in 'hosts' entries, the container names - I suppose it's authoritative for those, sort-of.

The glibc and musl-libc resolvers behave quite differently to each other, particularly with our ndots:0. Abstracting away those differences by pulling all requests up to our resolver would make things more consistent. (That'd mean not putting search domains into the container's resolv.conf. And, stopping glibc from looking at the dots in the hostname to invent a search domain, if possible). But, in some cases that might not be what people expect, and I guess behaviour would differ from other container environments.

So, I think it's all a bit subtle and use-case dependent. We need to make things more consistent and predictable, but any change is likely to result in breakage for someone. Hopefully we'll be able to find some sensible middle ground, without a proliferation of options that nobody quite understands!

from moby.

Yeah, the search is a tricky one, and may depend on the situation.

Either way; lots to look into

from moby.

Thanks for looking into this!

As a (rather surprising) workaround, please could you try adding --dns-search=. to your docker run command.

Yes, that seems to help:

❯ docker run --network web --dns-search=. debian cat /etc/resolv.conf
nameserver 127.0.0.11
options edns0 trust-ad ndots:0

As expected, this also makes resolving local domains in an alpine container work. AFAICT it cannot be configured at the network level, though? Still, a good workaround for now.

from moby.