The default values for the HTTP accept header are populated for the GET methods, but not the HEAD methods, leading to a divergence between the two method types.

Hello,

Currently, calling close() on DockerRegistryClientAsync instance renders that instance unable to initiate another aiohttp session. After calling close() the underlying session is closed but the variable reference for it is not set to None and subsequent calls to, eg., get_manifest fail with "Session is closed" error.

eg ImageName.parse('python')

Hello,

I've hit the problem of DRCA not refreshing tokens for repositories already interacted with. I find this as major problem, since from how DRCA is structured it's clear that DRCA is best instantiated once for the lifecycle of the application (taking care of connection pooling and credentials management) but with no token refreshing this is pretty much impractical now.

I have the use case where I'm downloading images for various forms of scanning that my backend is doing. To improve performance and scalability I'm checking if an image has already been scanned, so I can skip scanning it again. However, if the user supplies me with image:tag I have to resolve that to image@digest (by making a request via DRCA to the registry) and if I have indeed scanned that image this would mean I already acquired a token for that repository (scope). So, when I try to resolve a tag to digest a second time and this happens several minutes later, the token will be expired and the request would fail with 401 Unauthorized.

The same happens even with anonymous access to DockerHub because DockerHub mandates token-based access for anonymous users.

For the time being, I've made brutal hack to workaround the problem: monkey-patched DRCA with _get_token=_get_auth_token which forces getting new token on every request. Apparently, this is very bad for performance (and increases the chances of throttling on the registry side), so it'd be great to have token refresh logic.

Docker Hub now mandates going through the authentication process, even for anonymous accounts.

Please go through the authentication flow even if there's no credentials configured.

Hello again crashvb,

I'm writing this to suggest that token_based_endpoints config should be dropped and Token Authentication should be the default.

To support this suggestion I'll submit some facts as of today:

1. Some registries don't support Basic Authentication directly at all -- such as Docker.io and Quay.io. By directly, I mean to supply credentials to the registry itself, rather than to its Auth Server (eg. auth.docker.io/token). Docker registry documentation doesn't even mention that one can use Basic Auth to directly authenticate to a registry -- the only documented way is through the Token auth method.
2. There are some registry servers that support direct Basic Auth, such as Harbor. Howerver, the default (when no Authorization header is supplied) is token-based auth. This effectively means that every registry not explicitly known to DRCA (by setting explicit creds) will fail anonymous pull request. One interesting case is when trying to force token-auth for Harbor -- it won't work because of this line.
3. There are no registry server that I know of that don't support token-based auth.

One more related thing that I want to mark is the following use case: At our company we have several docker registries, but all of them have their user identities sync'ed. So, there might be many registries, but I can connect to all of them with the same credentials. Every team is free to spin up their registry for its own purposes, however it's very common for them to be interested in integrating with some automation tools/services that need to access the registry. With the current explicit credentials-per-endpoint approach, this automation tooling (that use DRCA) should explicitly know with what endpoints it will have to operate which is very limiting. It'd be great if there are some default creds to be used if no explicit configuration exist for this endpoint.

We should be able to test integration to a registry locally, without hammering Docker Hub and alike.

Hello crashvb,

I've encountered another interesting issue: At my company we're working with a lot of Artifactory-based registries and this uncovers a "bug" in the image resolution logic. The current logic is to always prefix images that don't contain "namespace" with library. However, Artifactory recommends to their users to move the "namespace" part of the repository as sub-domain part of the URL. In other words, usually one would expect to work with Artifactory as every other registry like so:
some-artifactory.com/library/python, but Artifactory restructures this into library.some-artifactory.com/python (the namespace will never be called library, but it's good for illustration purposes). Obviously, if DRCA parses that reference, it will modify it to library.some-artifactory.com/library/python which would return 404.

It was my impression that if image doesn't have repository "namespace" it should be assigned to library repository. Apparently, DRCA thinks the same. But it turns out that's not the case -- there is nothing in the Docker Registry spec about that and surprisingly docker pull command copes just fine with library.some-artifactory.com/python.

So, I've spend some hours looking through Go code of containerd and registry-handling code in Docker to see what's going on and found these:
containerd: https://github.com/containerd/containerd/blob/49a945b26b9d7bd485e49041b74aab3aab97c15a/reference/docker/reference.go#L666
Docker: https://github.com/distribution/distribution/blob/main/reference/normalize.go#L87
(They are the same)

So, it turns out that library should be only prefixed when working with DockerHub. Tomorrow I'll create a pull request with potential fix. Can you think of potential problems with that change?

There is a need to add support when executing through a proxy.

The process of getting an ImageName from get_tags() is annoying, especially if you were originally given an ImageName, not a string.

We should be able to chunk files using a generator provided to aiohttp, and calculate the size and digest at any point in time.

Hello crashvb,

Thanks for the great package, I'm happy that there is docker/oci registry client that works with python!

I've encountered a problem, though. aiodns package, used for DNS resolution in DRCA, has a shortcoming of returning either IPv4 or IPv6 addresses, but not both on DNS resolution. This causes domains having both A and AAAA records to be resolved by aiodns with only IPv6. Since IPv6 is not common yet, most OSes have working IPv6 stacks but no default route setup, which leads to OSError No Route to Host when trying get a blob, for example. This is especially true if the application using DRCA is running in Docker container, however, then the error is even more cryptic: instead of No Route to Host, it fails with Errno 99 [Cannot assign requested address].

For this exact problem, aiohttp switched the default resolver to ThreadedResolver from AsyncResolver (aio-libs/aiohttp@9fbb7d7).

The problem with IPv6 is coming from c-ares C lib, actually:
c-ares/c-ares#70

And it's tracked in 'aiodns', here:
saghul/aiodns#23

One workaround is to pair AsyncResolver with aiohttp.TCPConnector class that have an explicit family kwargs set to IPv4. Unfortunately, DRCA only supports passing custom arguments to AsyncResolver, but not to aiohttp.TCPConnector. So, the following hack should be made:

class DRCA_PATCHED(drca.DockerRegistryClientAsync):
    async def _get_client_session(self):
        if not self.client_session:
            self.client_session = aiohttp.ClientSession(
                connector=aiohttp.TCPConnector(
                    resolver=aiohttp.AsyncResolver(**self.resolver_kwargs),
                    ssl=self.ssl,
                    family=socket.AF_INET,
                )
            )
        return self.client_session

So, I guess, this will be encountered very often by users (and even more often when domains start to acquire AAAA records, I've encountered it with trying to download layers of python:3.9 image from docker.io), so maybe it will be better for DRCA to allow passing Connector options, or be able to switch the Resolver class.

There is no abstraction around manifests to get digests (in particular, config)

Hello it's me again! :)

I was just struck by a strange behaviour for one registry, returning HTTP 400 Bad Request on getting manifest. I've debugged the issue and it turned out the registry returns distribution manifest v1 and thus a slight typo in the default Accept header was causing the issue.
The problem is here (missing a comma on the line):
https://github.com/crashvb/docker-registry-client-async/blob/main/docker_registry_client_async/dockerregistryclientasync.py#L79

Add an option for retrying?

os.environ['DRCA_DEFAULT_REGISTRY'] = 'mirror.redacted.com'

from docker_registry_client_async import DockerRegistryClientAsync, FormattedSHA256, ImageName, Manifest

async def test():
    async with DockerRegistryClientAsync() as drca:
        name = ImageName.parse("busybox:1.30.1")
        result = await drca.get_manifest(name)
        manifest = result["manifest"].get_json()
        print(manifest)
        config_digest = FormattedSHA256.parse(manifest["config"]["digest"])
        mime = manifest['config']['mediaType']
        result = await drca.get_blob(name, config_digest, accept=mime)
        config = json.loads(result["blob"].decode("utf-8"))
        print(config)

asyncio.run(test())

$ ./python-pull.py 
{'schemaVersion': 2, 'mediaType': 'application/vnd.docker.distribution.manifest.v2+json', 'config': {'mediaType': 'application/vnd.docker.container.image.v1+json', 'size': 1496, 'digest': 'sha256:64f5d945efcc0f39ab11b3cd4ba403cc9fefe1fa3613123ca016cf3708e8cafb'}, 'layers': [{'mediaType': 'application/vnd.docker.image.rootfs.diff.tar.gzip', 'size': 755813, 'digest': 'sha256:53071b97a88426d4db86d0e8436ac5c869124d2c414caf4c9e4a4e48769c7f37'}]}
Traceback (most recent call last):
  File "./python-pull.py", line 75, in <module>
    asyncio.run(test())
  File "/usr/lib/python3.8/asyncio/runners.py", line 43, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.8/asyncio/base_events.py", line 608, in run_until_complete
    return future.result()
  File "./python-pull.py", line 71, in test
    result = await drca.get_blob(name, config_digest, accept=mime)
  File "/dogfood-home/eng_home/jbliss/ptc-testing/.venv/lib/python3.8/site-packages/docker_registry_client_async/dockerregistryclientasync.py", line 499, in get_blob
    data = await client_response.read()
  File "/dogfood-home/eng_home/jbliss/ptc-testing/.venv/lib/python3.8/site-packages/aiohttp/client_reqrep.py", line 973, in read
    self._body = await self.content.read()
  File "/dogfood-home/eng_home/jbliss/ptc-testing/.venv/lib/python3.8/site-packages/aiohttp/streams.py", line 358, in read
    block = await self.readany()
  File "/dogfood-home/eng_home/jbliss/ptc-testing/.venv/lib/python3.8/site-packages/aiohttp/streams.py", line 380, in readany
    await self._wait('readany')
  File "/dogfood-home/eng_home/jbliss/ptc-testing/.venv/lib/python3.8/site-packages/aiohttp/streams.py", line 296, in _wait
    await waiter
aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed

mirror.redacted.com is running registry:2 in mirror mode. Removing the environ set causes the problem to go away.