Cannot resolve DNS hostname of registry from docker-registry-browser about docker-registry-browser HOT 20 CLOSED

hi guys - ive had the same problem. I had my registry resolved as registry.gecko-is.co.uk:5000:[IP] but the registry browser could not resolve this. To fix it, i made sure the registry-browser was on the same overlay network, and used the docker-registry's internal ip address [10.x.x.x]:5000 as docker-registry-browser DOCKER_REGISTRY_URL param. Hope this helps someone else.

from docker-registry-browser.

Hi @mdebord1,

thanks for reporting that issue.

Since I'm not able to reproduce the issue currently (tried with adding your FQDN to my /etc/hosts - but of course that's not the same) I'd need some more information from you:

Would it be possible that you send me the output of docker inspect for the registry-browser container without the hardcoded IP? Also it would be good to know if the registry.myhost.local is resolved by an actual DNS server in your network or if you're relying on mDNS there.

I also had a look at the implementation of listing images and showing the details of a single one and they're using both the exact same HTTP client so it's really odd that you're experiencing that issue.

Thank you

Klaus

from docker-registry-browser.

Hi - Thanks for helping investigate.

• Yes it's DNS, a Bind9 DNS server on our internal network.

• Here is the docker inspect on the container USING the hostname: https://pastebin.com/7nf6gj5c

• Here is the resolv.conf of the container:

root@docker10z:/dockerdata/Docker-Compose/registry_browser_dns# docker exec -it -u root registry_browser_dns cat /etc/resolv.conf

# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 10.10.50.254
search myhost.local

• I did find something odd - when running nslookup from the container, it spits out "can't resolve '(null)' but then it ends up resolving.

root@docker10z:/dockerdata/Docker-Compose/registry_browser_dns# docker exec -it -u root registry_browser_dns sh

/app # nslookup registry.myhost.local
nslookup: can't resolve '(null)': Name does not resolve

Name:      registry.myhost.local
Address 1: 10.1.5.22 registry.myhost.local

/app # nslookup registry
nslookup: can't resolve '(null)': Name does not resolve

Name:      registry
Address 1: 10.1.5.22 registry

/app # ping registry.myhost.local
PING registry.myhost.local (10.1.5.22): 56 data bytes
64 bytes from 10.1.5.22: seq=0 ttl=64 time=0.043 ms

Please be aware that the domain name and hostnames have been changed for company privacy

from docker-registry-browser.

Thanks for the Pastebin 😀

I can't see anything special in that docker inspect output - it looks totally fine so it should not be a configuration issue on that level I think.

I did find something odd - when running nslookup from the container, it spits out "can't resolve '(null)' but then it ends up resolving.

That's ok. As far as I know the nslookup (at least the version from busybox in alpine) just tries to do a reverse lookup for the IP of your nameserver which to show the FQDN. I have the same warning in an blank alpine container:

[ meyer at km-macbookair in ~ ] ruby-2.4.2
» docker run -it alpine sh
/ # nslookup google.com
nslookup: can't resolve '(null)': Name does not resolve

Name:      google.com
Address 1: 216.58.207.174 muc11s04-in-f14.1e100.net
Address 2: 2a00:1450:4016:807::200e muc11s04-in-x0e.1e100.net

Currently I don't have any clue what could be the issue.
Is it always the case that the list of loads and the detail page raises this error or do you maybe also get the same error if you refresh the list in a faster sequence?

from docker-registry-browser.

Yea I'm baffled myself.

Here are the application logs after loading the front page, where the list of images loads successfully. In response to your question, I cannot seem to get this to fail. Even after multiple refreshes

I, [2017-12-21T18:15:41.232544 #7]  INFO -- : [481b85ad-369c-4afd-9e41-6965cc83496a] Started GET "/" for 10.10.60.254 at 2017-12-21 18:15:41 +0000
I, [2017-12-21T18:15:41.233193 #7]  INFO -- : [481b85ad-369c-4afd-9e41-6965cc83496a] Processing by RepositoriesController#index as HTML
I, [2017-12-21T18:15:41.234011 #7]  INFO -- : get https://registry.myhost.local:5001/v2/_catalog?n=100
D, [2017-12-21T18:15:41.234071 #7] DEBUG -- request: User-Agent: "Faraday v0.11.0"
I, [2017-12-21T18:15:41.261065 #7]  INFO -- Status: 200
D, [2017-12-21T18:15:41.261145 #7] DEBUG -- response: content-type: "application/json; charset=utf-8"
docker-distribution-api-version: "registry/2.0"
x-content-type-options: "nosniff"
date: "Thu, 21 Dec 2017 18:15:41 GMT"
content-length: "210"
connection: "close"
I, [2017-12-21T18:15:41.261760 #7]  INFO -- : [481b85ad-369c-4afd-9e41-6965cc83496a]   Rendering repositories/index.html.erb within layouts/application
I, [2017-12-21T18:15:41.263564 #7]  INFO -- : [481b85ad-369c-4afd-9e41-6965cc83496a]   Rendered repositories/index.html.erb within layouts/application (1.7ms)
I, [2017-12-21T18:15:41.264192 #7]  INFO -- : [481b85ad-369c-4afd-9e41-6965cc83496a]   Rendered application/_flash.html.erb (0.1ms)
I, [2017-12-21T18:15:41.264461 #7]  INFO -- : [481b85ad-369c-4afd-9e41-6965cc83496a] Completed 200 OK in 31ms (Views: 3.0ms)

I'm going to destroy everything and start from scratch and see what I can come up with.

from docker-registry-browser.

Ok after destroying and rebuilding the container fully from scratch I am now able to get the front page to fail with the same error. Upon refreshing several times, it will load, then it will fail again sporadically.

Very sporadic, might have something to do with my network but everything else resolves from DNS and it resolves at the command line without issue. I'm going to just keep using the hardcoded IP for now as a workaround.

from docker-registry-browser.

Thanks for the feedback.

I agree - it sounds very much like something in your network or maybe on the docker-host itself.
How up-to-date is your docker version?

from docker-registry-browser.

Can we re-open this issue?

I was browsing my registry and noticed some tags were missing. I tried another registry browser, and I was able to see the missing tags that do not appear in docker-registry-browser. I also verified this by using the CLI tool docker-ls to list all of my repositories and tags.

At first, I encountered similar problems to this issue. In my case, when I ran docker-registry-browser on a separate host from the registry, I had to use the IP address of the docker registry server to workaround the same DNS problem. Otherwise, the registry browser did not work at all.

However, when I ran docker-registry-browser on the registry server itself (using the hostname in the URL), I could browse the registry. It seemed like it was working until I realized I there were some tags missing.

But when I went back to using the IP address in the URL instead of the hostname, then all the tags appeared in the browser.

More info:
I encountered the same DNS error in the container logs: getaddrinfo: Name does not resolve

I started a shell inside the container and installed the bind-tools package. I was able to resolve the registry hostname via nslookup, but ping would say bad address. But, I could ping the registry if I used its IP address instead of the hostname.

from docker-registry-browser.

@jmmen1 could you please share a few details regarding the DNS setup you have there?
What kind of domain names are you using? Also the .local ones or something with a public domain name?

The part of the missing tags is strange. Did they simple miss in the list or is the problem visible on the tag-details page?

from docker-registry-browser.

The registry server is a linux vm running in my work's internal corporate network. I am connecting to the server from my corporate laptop via its FQDN, not a .local address.

The tags simply did not appear in the list. This was confirmed by another developer. He was working with some images, and he retagged some of his images locally, then pushed the new tags, but neither the old nor the new tags appeared in the list. It was if they did not exist at all.

I did notice that the majority of the tags that were missing were tags that contained numbers. For example, we have several tomcat images, and the image tagged "dev" appeared in the list, but the image tagged with just the version number, "7.0.53" did not appear in the list. Then, the developer pulled the 7.0.53 image, retagged it with the date, 7.0.53-01152019, then pushed it back to the registry, but that tag did not appear in the registry browser either.

from docker-registry-browser.

It seems to be something in the container. See my previous comment about installing the bind-tools package in the container. When running the container on a Mac host, it is consistent that ping will show the error "bad address". When I run the container on a Linux host, it's intermittent. Sometimes ping will work and other times it won't. I've tried with both host and bridge networking for the container.

When running the container on either a Mac or Linux host, and with either host or bridge networking, the "host" command is always able to resolve the registry server's IP address when given the FQDN, even if ping says "bad address".

from docker-registry-browser.

I tried to reproduce the issue(s) but wasn't able - sorry :(

What I have tried so far:

1. Mac running the application (without docker) talking to a registry container on the same host
2. Mac running the application (with docker) talking to a registry container on the same host
3. Mac running the application (with docker) talking to a registry running on a external Linux machine

For all three cases I used FQDNs provided by the router in my home network to see if I get any DNS issues. I also had a few different example images & tags on the registries to see if I can trigger the missing tags problem:

- <registry>/tomcat:dev
- <registry>/tomcat:7.0.53
- <registry>/tomcat:v7.0.53
- <registry>/something/tomcat:dev
- <registry>/something/tomcat:7.0.53
- <registry>/something/tomcat:v7.0.53

Maybe you could check and see if the DNS stuff is also not working in a plain ruby:2.6.0-alpine container?

Cheers, Klaus

from docker-registry-browser.

The issue seems to be in the base image.

I installed the bind-tools package in both a ruby:2.6.0-alpine container and an alpine:3.8 container. I encountered the same "bad address" error message when trying to ping a hostname. As before, pinging the IP address works.

from docker-registry-browser.

Just tried it as well with the base-image:

» docker run -it --rm ruby:2.6-alpine sh
/ # ping raspberrypi.fritz.box
PING raspberrypi.fritz.box (192.168.178.111): 56 data bytes
64 bytes from 192.168.178.111: seq=0 ttl=37 time=6.985 ms
64 bytes from 192.168.178.111: seq=1 ttl=37 time=12.491 ms
^C
--- raspberrypi.fritz.box ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 6.985/9.738/12.491 ms
/ # apk update; apk add bind-tools
fetch http://dl-cdn.alpinelinux.org/alpine/v3.8/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.8/community/x86_64/APKINDEX.tar.gz
v3.8.2-28-g785c236c3e [http://dl-cdn.alpinelinux.org/alpine/v3.8/main]
v3.8.2-28-g785c236c3e [http://dl-cdn.alpinelinux.org/alpine/v3.8/community]
OK: 9547 distinct packages available
(1/5) Installing libgcc (6.4.0-r9)
(2/5) Installing json-c (0.13.1-r0)
(3/5) Installing libxml2 (2.9.8-r1)
(4/5) Installing bind-libs (9.12.3-r0)
(5/5) Installing bind-tools (9.12.3-r0)
Executing busybox-1.28.4-r2.trigger
OK: 20 MiB in 35 packages
/ # nslookup raspberrypi.fritz.box
Server:		192.168.65.1
Address:	192.168.65.1#53

Non-authoritative answer:
Name:	raspberrypi.fritz.box
Address: 192.168.178.111

So it's not a global error with the base image but rather something only showing up in special cases.
I currently have no idea what could be the problem 😞

from docker-registry-browser.

It's definitely something with alpine. I ran a while loop in an alpine:edge container to ping the host once every 3 seconds, and the ping succeeded only one in 100 tries.

It never fails if I use an ubuntu container.

from docker-registry-browser.

Did you already try with the recently released alpine 3.9?

from docker-registry-browser.

I tried with alpine:edge, and alpine:3.9. If I run a loop of 100 pings, I rarely get a successful ping. Mostly, it's the "bad address" error.

However, if I create a network via "docker network create" and use that network in the "docker run" command, ping works perfectly every time.

from docker-registry-browser.

Ok thanks for testing.
I was just thinking that maybe they changed something in the new release which "fixed" your issue.

However since this seems to be an issue that only pops up in some special cases and also in the raw alpine image without my application I'd close this issue again.

What about the missing tags you mentioned - is this still a problem?

from docker-registry-browser.

I think the missing tags were related to the DNS issues. When I used the IP address in the URL, I can see all of the tags.

from docker-registry-browser.

Ok 👍

from docker-registry-browser.