Cannot use Github Container Registry about argocd-image-updater HOT 12 CLOSED

Can you try with tagsortmode: latest_first configuration option for your registry configuration, so basically

  registries.conf: |
    registries:
    - name: Github Container Registry
      api_url: https://ghcr.io
      ping: no
      prefix: ghcr.io
      credentials: pullsecret:argocd-image-updater/regcred
      tagsortmode: latest_first

This will prevent fetching additional metadata (which GHCR does not support). The tags returned by GHCR are already sorted by the date they were pushed to the registry.

Also, we should include the new prefix ghcr.io by default as alias for docker.pkg.github.com.

from argocd-image-updater.

Now the pod is in CrashLoopBackOff:

time="2020-09-25T05:10:40Z" level=info msg="argocd-image-updater v0.6.1 starting [loglevel:INFO, interval:2m0s, healthport:8080]"
time="2020-09-25T05:10:40Z" level=error msg="Could not load registry configuration from /app/config/registries.conf: unknown tag sort mode for registry Github Container Registry: latest_first"

My whole ConfigMap looks like:

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-image-updater-config
data:
  # Whether to use plain text connection (http) instead of TLS (https)
  argocd.plaintext: "true"
  registries.conf: |
    registries:
    - name: Github Container Registry
      api_url: https://ghcr.io
      ping: no
      prefix: ghcr.io
      credentials: pullsecret:argocd-image-updater/regcred
      tagsortmode: latest_first

Also I think that ghcr.io and docker.pkg.github.com should not be mixed up. They are both different registries with different implementations. ghcr.io is a new implementation which is current in public beta (see blog post) and was created due to limitations in docker.pkg.github.com (e.g. not being able to pull by digest). Finally ghcr.io will replace docker.pkg.github.com.

from argocd-image-updater.

I checked the code and I think it should be tagsortmode: latest-first with a hyphen rather than an underscore. When I use that I still get an error:

time="2020-09-25T05:30:17Z" level=error msg="Could not get tags from registry: unexpected manifest schema was received from registry: application/vnd.docker.distribution.manifest.v2+json (registry may not support the manifest type(s) you want: [application/vnd.docker.distribution.manifest.v1+prettyjws application/vnd.docker.distribution.manifest.v1+json])" alias=techwiki application=techwiki image_name=windsource/techwiki image_tag=latest registry=ghcr.io
time="2020-09-25T05:30:17Z" level=info msg="Starting image update cycle, considering 1 annotated application(s) for update"
time="2020-09-25T05:30:18Z" level=info msg="Processing results: applications=1 images_considered=1 images_skipped=0 images_updated=0 errors=1"
time="2020-09-25T05:30:18Z" level=error msg="Could not get tags from registry: unexpected manifest schema was received from registry: application/vnd.docker.distribution.manifest.v2+json (registry may not support the manifest type(s) you want: [application/vnd.docker.distribution.manifest.v1+prettyjws application/vnd.docker.distribution.manifest.v1+json])" alias=techwiki application=techwiki image_name=windsource/techwiki image_tag=latest registry=ghcr.io

from argocd-image-updater.

I checked the code and I think it should be tagsortmode: latest-first with a hyphen rather than an underscore.

Oops, you are right! Thank you, I will update the documentation (and also add a hint that this is required for GHCR).

When I use that I still get an error:

time="2020-09-25T05:30:17Z" level=error msg="Could not get tags from registry: unexpected manifest schema was received from registry: application/vnd.docker.distribution.manifest.v2+json (registry may not support the manifest type(s) you want: [application/vnd.docker.distribution.manifest.v1+prettyjws application/vnd.docker.distribution.manifest.v1+json])" alias=techwiki application=techwiki image_name=windsource/techwiki image_tag=latest registry=ghcr.io
time="2020-09-25T05:30:17Z" level=info msg="Starting image update cycle, considering 1 annotated application(s) for update"
time="2020-09-25T05:30:18Z" level=info msg="Processing results: applications=1 images_considered=1 images_skipped=0 images_updated=0 errors=1"
time="2020-09-25T05:30:18Z" level=error msg="Could not get tags from registry: unexpected manifest schema was received from registry: application/vnd.docker.distribution.manifest.v2+json (registry may not support the manifest type(s) you want: [application/vnd.docker.distribution.manifest.v1+prettyjws application/vnd.docker.distribution.manifest.v1+json])" alias=techwiki application=techwiki image_name=windsource/techwiki image_tag=latest registry=ghcr.io

Hm, I have just pushed an image with two tags to ghcr.io, made them public and used the following configuration for the image updater:

registries:
- name: Github Container Registry
  api_url: https://ghcr.io
  ping: no
  prefix: ghcr.io
  tagsortmode: latest-first

When I run the updater on an application configured to use this image and annotated it correctly, it seems successful:

INFO[0240] Starting image update cycle, considering 1 annotated application(s) for update 
DEBU[0240] Processing application helm-guestbook        
DEBU[0240] Considering this image for update             alias=gb application=helm-guestbook image_name=jannfis/ks-guestbook-demo image_tag=0.1 registry=ghcr.io
DEBU[0240] Using version constraint '>=0.1' when looking for a new tag  alias=gb application=helm-guestbook image_name=jannfis/ks-guestbook-demo image_tag=0.1 registry=ghcr.io
DEBU[0241] found 2 from 2 tags eligible for consideration  image="ghcr.io/jannfis/ks-guestbook-demo:0.1"
INFO[0241] Setting new image to ghcr.io/jannfis/ks-guestbook-demo:0.2  alias=gb application=helm-guestbook image_name=jannfis/ks-guestbook-demo image_tag=0.1 registry=ghcr.io
DEBU[0241] target parameters: image-spec= image-name=image.repository, image-tag=image.tag  application=helm-guestbook image=ghcr.io/jannfis/ks-guestbook-demo
INFO[0241] Successfully updated image 'ghcr.io/jannfis/ks-guestbook-demo:0.1' to 'ghcr.io/jannfis/ks-guestbook-demo:0.2', but pending spec update (dry run=true)  alias=gb application=helm-guestbook image_name=jannfis/ks-guestbook-demo image_tag=0.1 registry=ghcr.io
INFO[0241] Dry run - not performing spec update          application=helm-guestbook
INFO[0241] Processing results: applications=1 images_considered=1 images_skipped=0 images_updated=1 errors=0 

Can you please provide the image updater related annotations you used on your application, they might be the missing piece to the puzzle for reproducing the issue? Thanks!

from argocd-image-updater.

I think I have found the bug and fixed it with #93 - will prepare a v0.6.2 release containing the fix later this day

from argocd-image-updater.

I use a private repo on ghcr.io with commit hashes as tags like ghcr.io/windsource/myrepo:b8b7c74. One tag is the 'latest' tag.

The application looks like:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  annotations:
    argocd-image-updater.argoproj.io/image-list: myrepo=ghcr.io/windsource/myrepo
    argocd-image-updater.argoproj.io/myrepo.update-strategy: latest
  name: myapp
  namespace: argocd
spec:
  destination:
    namespace: default
    server: https://kubernetes.default.svc
  project: default
  source:
    path: myapp
    repoURL: https://github.com/windsource/myapps.git
    targetRevision: HEAD
  syncPolicy:
    automated: {}

from argocd-image-updater.

I use a private repo on ghcr.io with commit hashes as tags like ghcr.io/windsource/myrepo:b8b7c74. One tag is the 'latest' tag.

In this case, I would also suggest to use a tag match annotation - because otherwise, latest will be considered a valid tag and might end up as the target tag to use (which you probably wouldn't like to).

I think an annotation such as argocd-image-updater.argoproj.io/myrepo.tag-match: regexp:^[0-9a-f]+$ would be the right one for your tag naming strategy, so that only tags matching your git commit hashes naming will be considered for upgrade.

Maybe it would also make sense to provide a new annotation, such as argocd-image-updater.argoproj.io/<repo>.ignore-tags that lets the user ignore certain tags such as latest without having to use a tag-match. I guess this could be something for v0.7.

from argocd-image-updater.

Hey, please check if v0.6.2 resolves your problem.

from argocd-image-updater.

Hi, thanks! Now I do not get error messages anymore. The first time the image has been updated but other changes to the repo are not applied to my application. I removed the 'latest' tag from the repo to avoid confusion. Thanks for the hit with tag matches.

The logs when starting argocd-image-updater with the image not being up-to-date now looks like:

time="2020-09-25T10:07:49Z" level=info msg="argocd-image-updater v0.6.2 starting [loglevel:INFO, interval:2m0s, healthport:8080]"
time="2020-09-25T10:07:49Z" level=info msg="Loaded 1 registry configurations from /app/config/registries.conf"
time="2020-09-25T10:07:49Z" level=info msg="ArgoCD configuration: [server=argocd-server.argocd, auth_token=true, insecure=false, grpc_web=false, plaintext=true]"
time="2020-09-25T10:07:49Z" level=info msg="Starting health probe server TCP port=8080"
time="2020-09-25T10:07:49Z" level=info msg="Warming up image cache"
time="2020-09-25T10:07:50Z" level=info msg="Finished cache warm-up, pre-loaded 0 meta data entries from 5 registries"
time="2020-09-25T10:07:50Z" level=info msg="Starting image update cycle, considering 1 annotated application(s) for update"
time="2020-09-25T10:07:50Z" level=info msg="Processing results: applications=1 images_considered=1 images_skipped=0 images_updated=0 errors=0"
time="2020-09-25T10:09:50Z" level=info msg="Starting image update cycle, considering 1 annotated application(s) for update"
time="2020-09-25T10:09:51Z" level=info msg="Processing results: applications=1 images_considered=1 images_skipped=0 images_updated=0 errors=0"
time="2020-09-25T10:11:51Z" level=info msg="Starting image update cycle, considering 1 annotated application(s) for update"

from argocd-image-updater.

I have tested a little, and it seems that ghcr.io returns the list of tags in opposite order than docker.pkg.github.com does - so actually, if you set tagsortmode: latest-last, it should work as expected. At least it did in my tests.

Unfortunately, both GitHub registry implementations don't return a v1 manifest for the image tags, which is the only manifest version that contains information about the creation date of the image, so we must rely on the order of the tags returned.

Probably it will also a good idea to implement some kind of registry inspection command one can use to find out such things.

from argocd-image-updater.

Now it works! Thanks a lot for the great support!

from argocd-image-updater.

I guess we can close this issue. Please re-open if your issue persists.

from argocd-image-updater.