Custom registry, push error on self-signed cert about buildx HOT 23 CLOSED

A marginally more robust work-around, but still not pretty (no error checking etc):

BUILDER=$(sudo docker ps | grep buildkitd | cut -f1 -d' ')
sudo docker cp YOUR-CA.crt $BUILDER:/usr/local/share/ca-certificates/
sudo docker exec $BUILDER update-ca-certificates
sudo docker restart $BUILDER

A possible solution/suggestion would be to allow for something like:
docker secret create a-ca-secret my-ca-file.crt
docker buildx create ---name builder --driver-opt private-ca-cert=/run/secrets/a-ca-secret
...and then have the e.g. the docker-container builder pull in the cert and call update-ca-certificates when it starts.

from buildx.

I managed to work-around this - by adding my own CA's cert to the generated container, and restarting it.

$ docker ps|grep 'moby/buildkit'
ee110c9e6dfc        moby/buildkit:buildx-stable-1   "buildkitd"              7 minutes ago       Up 7 minutes                                                                                       buildx_buildkit_mybuilder0

$ docker exec -it ee110c9e6dfc sh
$$ cat >> /etc/ssl/certs/ca-certificates.crt <<'EOF'
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
EOF
$$ exit

$ docker restart ee110c9e6dfc

It aint pretty, but it works!

from buildx.

Considering this issue has been open for over two years, I was just curious if there is an update and/or plans to address?

from buildx.

as I just bumped into this issue, I'd like to add one more option instead of mangling an existing builder container. driver-opt has a image option (for docker-container driver).

I went with the option to have a 2 line Dockerfile that adds my internal CAs to moby/buildkit and use that image when creating the builder.

from buildx.

can we make buildx skip verify private registry cert like docker ( /etc/docker/daemon.json insecure-registries) ？ then we have no need to provide CA cert for buildx instance.

from buildx.

Reboot fixed it for me, I'm not sure why..

from buildx.

A marginally more robust work-around, but still not pretty (no error checking etc):

BUILDER=$(sudo docker ps | grep buildkitd | cut -f1 -d' ')
sudo docker cp YOUR-CA.crt $BUILDER:/usr/local/share/ca-certificates/
sudo docker exec $BUILDER update-ca-certificates
sudo docker restart $BUILDER

A possible solution/suggestion would be to allow for something like:
docker secret create a-ca-secret my-ca-file.crt
docker buildx create ---name builder --driver-opt private-ca-cert=/run/secrets/a-ca-secret
...and then have the e.g. the docker-container builder pull in the cert and call update-ca-certificates when it starts.

Tried that, but it didn't work.

from buildx.

Finally, we solved this issue. And there are two steps we need to confirm before we use to build:

• Make sure we use ca.crt was created by Harbor(we found the correct pah of it is /home/harbor/ca/ca.crt), (it is not the one that is related to Harbor's domain).
• If you are in the CentOS (yum) distribution GNU/Linux environment, make sure the host trusts Harbor's certification

# Execution of this command on the host
cp /home/harbor/ca/ca.crt /etc/pki/ca-trust/source/anchors/ update-ca-trust extract

All things are good to go.

from buildx.

can we make buildx skip verify private registry cert like docker ( /etc/docker/daemon.json insecure-registries) ？ then we have no need to provide CA cert for buildx instance.

from buildx.

can we make buildx skip verify private registry cert like docker ( /etc/docker/daemon.json insecure-registries) ？ then we have no need to provide CA cert for buildx instance.

from buildx.

In my case, it wasn't a self-signed cert but a company-wide CA cert. It's very possible that CA cert wasn't present in the environment I was running buildx from though so I could look into that.

from buildx.

I'm seeing similar with a standalone registry server and locally created CA/certificates. I'm able to run a docker login after adding a /etc/docker/certs.d/<hostname>:5000/ca.crt file, but buildx doesn't appear to use this.

from buildx.

@fopina can you elaborate on how you did that? I have a 2 line dockerfile where I copy a cert into /usr/local/share/ca-certificates/, and I can run it and exec into it to verify that it's there. I create the image locally, tagging it like testbuilder:latest

When I run docker buildx create --name tester --builder testbuilder:latest, the resulting container that's spun up doesn't have the cert in that folder:

Create and verify the image:

% docker run --rm -ti testbuilder sh
INFO[0000] auto snapshotter: using native               
WARN[0000] using host network as the default            
INFO[0000] found worker "t3jdqv8o0ov117tdipqjmfnly", labels=map[org.mobyproject.buildkit.worker.executor:oci org.mobyproject.buildkit.worker.hostname:000f6536acfa org.mobyproject.buildkit.worker.snapshotter:native], platforms=[linux/amd64 linux/arm64 linux/riscv64 linux/ppc64le linux/s390x linux/386 linux/arm/v7 linux/arm/v6] 
WARN[0000] skipping containerd worker, as "/run/containerd/containerd.sock" does not exist 
INFO[0000] found 1 workers, default="t3jdqv8o0ov117tdipqjmfnly" 
WARN[0000] currently, only the default worker can be used. 
INFO[0000] running server on /run/buildkit/buildkitd.sock

# In a different terminal window:
% docker ps
CONTAINER ID   IMAGE         COMMAND          CREATED          STATUS          PORTS     NAMES
000f6536acfa   testbuilder   "buildkitd sh"   18 seconds ago   Up 17 seconds             jovial_blackwell

% docker exec -it jovial_blackwell sh
/ # cd /usr/local/share/ca-certificates/
/usr/local/share/ca-certificates # ls
certs.pem

using it as a builder?

% docker buildx create --name tester --builder testbuilder:latest
% docker buildx use tester
% docker buildx inspect --bootstrap
[+] Building 1.0s (1/1) FINISHED                                                                                                       
 => [internal] booting buildkit                                                                                                   1.0s
 => => pulling image moby/buildkit:buildx-stable-1                                                                                0.4s
 => => creating container buildx_buildkit_tester0                                                                                 0.6s
Name:   tester
Driver: docker-container

Nodes:
Name:      tester0
Endpoint:  unix:///var/run/docker.sock
Status:    running
Platforms: linux/amd64, linux/arm64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6

% docker ps
CONTAINER ID   IMAGE                           COMMAND       CREATED         STATUS         PORTS     NAMES
d9c5e77418fc   moby/buildkit:buildx-stable-1   "buildkitd"   9 seconds ago   Up 8 seconds             buildx_buildkit_tester0

docker exec -ti d9c5e77418fc sh
/ # ls /usr/local/share/ca-certificates/
/ # %                                                     < -- nothing here

from buildx.

@bdharrington7 have you tried curl in that image (on the registry endpoint) to validate that they have been properly installed?
I believe you have to run update-ca command unless you already pushed the total file.

from buildx.

I didn't try curl, but I did try using the --push option when I built another image using buildx, and it still failed, because the registry I'm pushing to uses an internal cert. I assumed that the image I specified with the --builder flag wasn't being used, since it didn't have the same information in the one I actually built when I spun up testbuilder:latest manually (the cert I baked into the testbuilder image at /usr/local/share/ca-certificates/ isn't even there)

from buildx.

I didn't try curl, but I did try using the --push option when I built another image using buildx, and it still failed, because the registry I'm pushing to uses an internal cert. I assumed that the image I specified with the --builder flag wasn't being used, since it didn't have the same information in the one I actually built when I spun up testbuilder:latest manually (the cert I baked into the testbuilder image at /usr/local/share/ca-certificates/ isn't even there)

To modify buildx builder image use buildx create --driver-opt image=yourimage

from buildx.

@tonistiigi is there any documentation on how to get thos to work now?

from buildx.

@ravensorb https://github.com/docker/buildx/blob/master/docs/guides/custom-registry-config.md

from buildx.

Hi, guys. We still hit this issue in buildx v0.8.1

➜  multi-build-template git:(master) ✗ docker buildx version
github.com/docker/buildx v0.8.1-docker 5fac64c2c49dae1320f2b51f1a899ca451935554

➜  multi-build-template git:(master) ✗ make build-demo-amd64  
docker buildx build --builder demo --platform linux/amd64 \
-o type=docker \
-t demo/demo:v0.1-amd64  \
-f ./Dockerfile.amd64 \
--build-arg GOARCH=amd64 \
--build-arg GO_VERSION=1.13.15 \
--build-arg CGO_ENABLED=0 \
--build-arg GO111MODULE=on \
--build-arg APPLICATION_NAME=demo \
--build-arg FILE_NAME=./demo.go \
--build-arg COMMIT_ID=$(git rev-parse --short HEAD)+"dirty" .
[+] Building 0.2s (7/7) FINISHED                                                               
 => [internal] load .dockerignore                                                         0.0s
 => => transferring context: 2B                                                           0.0s
 => [internal] load build definition from Dockerfile.amd64                                0.0s
 => => transferring dockerfile: 915B                                                      0.0s
 => ERROR [internal] load metadata for harbor.mysql.dbdns.repo-name.cn/library/ubi8@sh  0.2s
 => ERROR [internal] load metadata for harbor.mysql.dbdns.repo-name.cn/library/golang:  0.2s
 => [auth] library/golang:pull token for harbor.mysql.dbdns.repo-name.cn                0.0s
 => [auth] library/ubi8:pull token for harbor.mysql.dbdns.repo-name.cn                  0.0s
 => [auth] library/ubi8:pull token forharbor.mysql.dbdns.repo-name.cn                  0.0s
------
 > [internal] load metadata for harbor.mysql.dbdns.repo-name.cn/library/ubi8@sha256:fef1f0adb231b94c80f22be55bb7678b5c8898d3eb314ee7ceeae7c520d76958:
------
------
 > [internal] load metadata for harbor.mysql.dbdns.repo-name.cn/library/golang:1.13.15:
------
Dockerfile.amd64:2
--------------------
   1 |     ARG GO_VERSION
   2 | >>> FROMharbor.mysql.dbdns.repo-name.cn/library/golang:$GO_VERSION as gobuilder
   3 |     
   4 |     ARG CGO_ENABLED
--------------------
error: failed to solve: failed to fetch oauth token: Post "https://harbor.mysql.dbdns.repo-name.cn/service/token": x509: certificate signed by unknown authority
make: *** [build-demo-amd64] Error 1

Here is the configuration of buildkit.toml

debug=true

[registry."otherrepo-workfine.io:5000"]
  http = true
  insecure = true

[registry."harbor.mysql.dbdns.repo-name.cn"]
  ca=["/etc/docker/certs.d/harbor.mysql.dbdns.repo-name.cn/ca.crt"]
  [[registry."harbor.mysql.dbdns.repo-name.cn".keypair]]
    key="/etc/docker/certs.d/harbor.mysql.dbdns.repo-name.cn/harbor.mysql.dbdns.repo-name.cn.key"
    cert="/etc/docker/certs.d/harbor.mysql.dbdns.repo-name.cn/harbor.mysql.dbdns.repo-name.cn.cert"

from buildx.

We checked all the certifications are copied into the buildx container however, it looks like did not work.

from buildx.

More information about the token, so, we believe the token is working well.

# Get token
curl -i -k -u admin:cddHarbor@1234 'https://harbor.mysql.dbdns.repo-name.cn/service/token?service=harbor-registry&scope=repository:library/golang:pull,push'

# Get information successfully from the harbor with `ca.crt` and `token` we got above.
curl -i --cacert /home/harbor/ca/ca.crt  -H "Content-Type: application/json" -H "Authorization:  Bearer `tokenStr`" \
 'https://harbor.mysql.dbdns.repo-name.cn/v2/library/golang/manifests/1.15.15'

from buildx.

I checked all the certifications are copied into the buildx container and "cp /home/harbor/ca/ca.crt /etc/pki/ca-trust/source/anchors/ update-ca-trust extract". however, it looks like did not work.

from buildx.

I'm locking this thread as this has been solved and recent comments are not related.

from buildx.