Building ARM images ;) about docker-buildbox HOT 10 CLOSED

the most used images are now available for arm64. I'll close this meta issue and track the remaining work in dedicated issues :

• #965
• #966
• #967
• #968
• #969
• #970
• #971
• #972

from docker-buildbox.

The correct way should be to build the image on multiple architectures using docker buildx, with one tag pointing to multiple arch, then letting docker pull the correct image depending on the user's architecture.

Buildx doc : https://docs.docker.com/buildx/working-with-buildx/

A blog post about it : https://www.docker.com/blog/multi-arch-images/

The issue is I don't think our current python lib docker-py supports it.

from docker-buildbox.

It seems like we'll have to use the legacy way to build multiple architectures (one tag = one architecture) if we use our current lib docker-py : It doesn't support BuildKit.

The lib allows to pass a platform when building but that's it : https://docker-py.readthedocs.io/en/stable/images.html

So we'll have to build and push each platform separately, then once it's done we'll be able to create a multi-arch tag with docker manifest or its docker-py equivalent.

The docker CLI equivalent of this operation would be :

# AMD64
$ docker build -t ekino/ci-node:latest-amd64 --build-arg ARCH=amd64/ .
$ docker push ekino/ci-node:latest-amd64

# ARM64V8
$ docker build -t ekino/ci-node:latest-arm64v8 --build-arg ARCH=arm64v8/ .
$ docker push ekino/ci-node:latest-arm64v8

$ docker manifest create \
ekino/ci-node:latest \
--amend ekino/ci-node:latest-amd64 \
--amend ekino/ci-node:latest-arm64v8 \

With ARCH being used in the dockerfile to pull the correct base image.
We could also duplicate the Dockerfiles + have dedicated jobs for each step, that would allow dependabot to upgrade the base images.

The docker manifest step is of course optional, but IMO having an intel tag, an arm tag, and a default tag pointing to intel would not be acceptable. By the way, docker-py doesn't support manifest either. 😢

We could also switch to https://gabrieldemarmiesse.github.io/python-on-whales/, which sits on top of the native docker CLI in Go instead of reimplementing it. It would allow us to use buildx, but would still require the base image to have multiple architectures available behind the tag we're using. Most of them do anyway, we're talking about official OS images. 😄

See https://www.docker.com/blog/multi-arch-build-and-images-the-simple-way/ for more information

cc @ekinoben, if you have time to work on this issue, I'll be happy to help.

from docker-buildbox.

So, we need to remove the client call, to call the docker command directly using the build sub-command.

from docker-buildbox.

I did a quick audit of our base images - see buildbox-image-audit.20211207.xlsx

It looks like they all support multi-arch builds (at minimum, amd64 & arm64, which I believe is sufficient for us?) except for box react-native, which is based on openjdk:8u282-jdk-slim and only supports amd64.

That base image looks to be a bit out of date anyway. Later versions (e.g. openjdk:18-jdk-slim) support both amd64 & arm64 - can we update react-native to use this later base image? This would save us having to pass an ARCH arg into each Dockerfile, or use arch-specific Dockerfiles.

from docker-buildbox.

Great work @ekinoben !

I think we just need to make sure the android sdk is compatible with the version of java we use. I'll contact Ced and Lucas to get some info (I'll add you to the chat).

Maybe they don't use it anymore, idk.

Jdk 18 would be too bleeding edge anyway, 11 or 17 would be my guess.

from docker-buildbox.

As seen with Lucas, Ced, and @ekinoben, the react-native image isn't used by our mobile teams anymore (and isn't up to date anyway).

They switched to mobile-specific runners for build and deploy.

I'll create an issue to delete it.

from docker-buildbox.

@ekinoben react-native is no more :)

from docker-buildbox.

Quick progress update: I've switched the Python build script to use python-on-whales / docker buildx, and successfully built some multi-arch images. I've also updated all the affected Dockerfiles to fetch architecture-specific dependencies, using the TARGETARCH env var that buildx populates (i.e. replaced hardcoded references to x86 / amd64).

I'm currently trying to solve the "how to test it" issue: buildx with a multi-arch image won't give you a usable local image. It'll be in the build cache, but you can't run it, so we can't test it. As I see it, we could:

• add --push arg to docker buildx build. This will build and push the image to the registry, which means pushing to the registry before testing it. Which sounds kind of bad.

• build for each arch individually, test locally (QEMU?) then combine via docker manifest create and push to registry. This kind of defeats the point of using buildx.

• build for each arch individually, test locally (QEMU?), then build for all arch's using buildx, and push to registry. This means building each image twice.

• build multi-arch locally via buildx, export as tar (i.e. docker buildx build -o - - > image_name.tar) and load it via docker load for testing. Then push the tar to the repo? Or... something.

Any thoughts on the above?

from docker-buildbox.

Hi @ekinoben, sorry for the delay.

I didn't know this issue, that's too bad.

IMO the most sensible option would be to use docker manifest create, which should work through python-on-whales, since it only build images once, test the images that we'll push, and just add on "merge" step. Too bad for buildx. :(

For testing, maybe we can keep the existing x86-64 tests right now, and think about a clean solution later ? That would allow M1 Mac users to use our arm images faster, give us some feebacks etc.

from docker-buildbox.