problem deleting image or tag 16.6.1 about docker-gitlab HOT 13 OPEN

Hi @sachilles @avvertix,

I just tried out the simple following procedure:

• Starting from a registry:2.8, I replaced the image by the latest gitlab-registry registry.gitlab.com/gitlab-org/build/cng/gitlab-container-registry:v3.88.0-gitlab
• I changed registry mounted folder ownership to 1000:1000 and then started the registry and tried building an OCI image.
• Everything worked like a charm, I can now push/pull images from images and from my dev box as well as delete images from the gitlab interface or sidekiq jobs.

# docker-compose.yml
# [...]
  registry:
    #image: registry:2.8
    image: registry.gitlab.com/gitlab-org/build/cng/gitlab-container-registry:v3.88.0-gitlab
    restart: unless-stopped
    expose:
      - "5000"
    ports:
      - "127.0.0.1:5000:5000"
    volumes:
      - ./registry-config.yml:/etc/docker/registry/config.yml # this has changed as described before by @avvertix 
      - /opt/gitlab/disk/data/shared/registry:/var/lib/registry/docker/registry
      - /etc/ssl/private/registry.my-registry.com:/certs
    environment:
      REGISTRY_LOG_LEVEL: info
      REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY: /var/lib/registry
      REGISTRY_AUTH_TOKEN_REALM: https://my-gitlab.com/jwt/auth
      REGISTRY_AUTH_TOKEN_SERVICE: container_registry
      REGISTRY_AUTH_TOKEN_ISSUER: gitlab-issuer
      REGISTRY_AUTH_TOKEN_ROOTCERTBUNDLE: /certs/cert.crt
      REGISTRY_STORAGE_DELETE_ENABLED: "true"
    networks:
      - web
# [...]

Hope this will help.

PS: I obviously made a backup before doing the operation.

from docker-gitlab.

I actually stumbled on https://gitlab.com/gitlab-org/container-registry/-/issues/958#note_1471217687 that consider my setup (using registry:2.7.1) as a third party registry that will not be supported in the future.

In that thread is suggested to switch to the gitlab-container-registry version as declared in the Omnibus package. Apparently for Gitlab 16.6.2 the gitlab-container-registry version is v3.86.1-gitlab.

I tried the switch

1. I made a copy of the configuration file /etc/docker/registry/config.yml (apparently without an initial configuration file it won't start)
2. Replaced the registry docker image and mounted the new configuration file

   registry:
-    image: registry:2.7.1
+    image: "registry.gitlab.com/gitlab-org/build/cng/gitlab-container-registry:v3.86.1-gitlab"
     volumes:
+      - ./registry:/etc/docker/registry
       - /data/gitlab/registry:/registry
       - ./certs:/certs

I experienced an issue with file system permission during deletion, but I haven't figured out the correct user to switch to. I indeed temporary solved it by allowing write to everyone.

from docker-gitlab.

https://gitlab.com/gitlab-org/gitlab/-/issues/423459 that the suggested way is to migrate the entire contents of the Docker registry by using the GitLab container registry

Dear @sachilles I also saw the link you shared, but my interpretation is that the issue refers to the migration from filesystem metadata (or object storage) to the database storage for the images metadata. As far as I understood migrating to the database metadata is in beta and not yet fully complete that's why I didn't try that way.

I saw a workaround following the feedback issue on the use of external registries.

I totally agree that my approach is just a workaround, but probably could make easier the migration path until all official migrations tools are available.

from docker-gitlab.

Hi,

We are facing the same issue. In addition to the reported details, the gitlab registry cleanup policy when enabled will generate plenty of logs in both sides due to images not being deleted.

We got several instances, but the one we are facing this issue in is using registry 2.8.x instead of 2.7.x.

It seems related to several things :

• Gitlab has deprecated external registry support in 16.x

• Such an issue has been reported before

  • Gitlab uses image deletion by manifest which needs to be enabled on the registry
  • But it seems something has changed between registry 2.7.x and 2.8.x since gitlab uses an url like /v2/<repository>/manifests/<tag> in 16.6.2 and /v2/<repository>/manifests/<digest without sha256 prefix> making the deletion fail

  Is there any documented way to use gitlab registry with sameersbn image ?

from docker-gitlab.

Hi,

unfortunately I've been using only the integration of the docker registry into GitLab as pointed out in https://github.com/sameersbn/docker-gitlab/blob/master/docs/container_registry.md#gitlab-container-registry.

However, following https://docs.gitlab.com/ee/administration/packages/container_registry.html#self-compiled-installations, the integrated registry could be activated by using another image.

Did anyone try this?

from docker-gitlab.

Hi @sachilles,

unfortunately I've been using only the integration of the docker registry into GitLab as pointed out in https://github.com/sameersbn/docker-gitlab/blob/master/docs/container_registry.md#gitlab-container-registry.

This is what we did, didn't you met the issue we are facing? Did you try to delete container images from gitlab interface or using the cleanup policy?

However, following https://docs.gitlab.com/ee/administration/packages/container_registry.html#self-compiled-installations, the integrated registry could be activated by using another image.

Did anyone try this?

I didn't have time to try this out yet, I wanted to try using the integrated gitlab registry, directly from the main gitlab image by passing the registry disk to the main service, but I'm not sure it will work out of the box :\

from docker-gitlab.

I'm also experiencing errors when deleting images from the container registry. I noticed the problem after upgrading to 16.6. I suppose it is connected to Gitlab forking the Docker distribution registry. In my case I'm currently using registry:2.7.1 from Docker Distribution to host container registry images. @ymazzer @sachilles are you using the same version?

from docker-gitlab.

Hi @ymazzer,

This is what we did, didn't you met the issue we are facing? Did you try to delete container images from gitlab interface or using the cleanup policy?

I have the same problem. It doesn't matter if I try to delete container images via the GitLab web interface or via the cleanup policy. The result is the same. (I'm using the "official" self-hosted docker registry in version 2.8.1.)

I didn't have time to try this out yet, I wanted to try using the integrated gitlab registry, directly from the main gitlab image by passing the registry disk to the main service, but I'm not sure it will work out of the box :\

Okay, I guess the proposed way is to migrate the content of the "official" self-hosted docker registry (see https://gitlab.com/gitlab-org/gitlab/-/issues/423459).

from docker-gitlab.

Dear @avvertix,

I found somewhere (see https://gitlab.com/gitlab-org/gitlab/-/issues/423459) that the suggested way is to migrate the entire contents of the Docker registry by using the GitLab container registry, as several changes were included after the container registry was forked by the GitLab maintainers. (My first attempt to replace the Docker registry with the GitLab container registry failed).

However, please make sure to backup the entire Docker registry data directories before the necessary migration.

Once I (or someone else) is successful, we will update the corresponding configuration files and documentation.

from docker-gitlab.

@avvertix,
This solution solved the problem and related to deleting the images.
Tested on version 16.7.0. however I had to clean all the images within the registry.

Thanks

from docker-gitlab.

however I had to clean all the images within the registry.

@Gabrielandre02 could you tell me more about why you had to clean all the images, as in the end I was able to use the same storage directory by ensuring that the user within the docker image had access to the filesystem. (btw seems that in my case the user on the host os didn't have write permission on a parent folder in the storage, after I ensured that all the folders/files are writeable by the current user everything worked without changing nothing)

from docker-gitlab.

Will there be an update of the docs and an “official” way for migrating to the new docker image?

from docker-gitlab.

@etlam Yes, the project-related docs will be updated possible.

from docker-gitlab.