Giter VIP home page Giter VIP logo

Comments (7)

thaJeztah avatar thaJeztah commented on September 20, 2024 1

Both targets should be built.

Hm, I can double-check with the BuildKit maintainers, but I don't think the content of the mount should make a difference w.r.t. the build-cache; the expectation for cache is that the RUN command cannot depend on any state of the cache prior to the RUN command being executed, so the id should make no difference, other than to help with concurrency (two RUN instructions trying to modify content of the same mounts). https://docs.docker.com/reference/dockerfile/#run---mounttypecache

Contents of the cache directories persists between builder invocations without invalidating the instruction cache. Cache mounts should only be used for better performance. Your build should work with any contents of the cache directory as another build may overwrite the files or GC may clean it if more storage space is needed.

In other words, --mount type=cache is not taken into account for the build-cache, and given that both apt-cache update instructions start from the same base-image, both could share the same build-cache.

from moby.

thaJeztah avatar thaJeztah commented on September 20, 2024

In general, I would definitely recommend combining apt-get update with apt-get install into a single RUN though, so that updating the list of packages to install forces the cache to be invalidated (otherwise the apt-get update may never be updated)

from moby.

thaJeztah avatar thaJeztah commented on September 20, 2024

Had a quick chat on Slack, and it looks like there was a previous bug that caused the id to cause the cache to be invalidated; this was a bug, so it's possible you were depending on that bug here;

from moby.

vitoyucepi avatar vitoyucepi commented on September 20, 2024

If it was a bug, it was an absolutely amazing one. For me, it solves the problem of building multiple images that look the same on the local machine at the same time. In CI, I build each image in a separate namespace so they don't overlap.

Here are some examples

2.2/Dockerfile 
FROM alpine:3.19.1@sha256:c5b1261d6d3e43071626931fc004f70149baeba2c8ec672bd4f27761f8e1ad6b as downloader

# renovate: datasource=github-tags depName=savonet/liquidsoap-release-assets
ARG LIQUIDSOAP_VERSION=2.2.4

ARG LIQUIDSOAP_RELEASE="https://github.com/savonet/liquidsoap-release-assets/releases/download/v${LIQUIDSOAP_VERSION}"
ARG LIQUIDSOAP_DEB="liquidsoap_${LIQUIDSOAP_VERSION}-ubuntu-jammy-2_amd64.deb"
ARG LIQUIDSOAP_SHA512SUM="d16026b57ca9fbca6f89e1ba8523c09b6693868607c56119769cb0a3303b4328978f5e0a62ed9ea509bebb2fa5935a24cb03c6836c139cc2dae0ad55fdb6b036"

# hadolint ignore=DL3047,DL4006
RUN set -eux; \
      mkdir /packages; \
      wget "${LIQUIDSOAP_RELEASE}/${LIQUIDSOAP_DEB}" -O /packages/liquidsoap.deb; \
      echo "${LIQUIDSOAP_SHA512SUM}  /packages/liquidsoap.deb" | sha512sum -c -;

FROM ubuntu:22.04@sha256:77906da86b60585ce12215807090eb327e7386c8fafb5402369e421f44eff17e

ENV TZ=UTC
ARG DEBIAN_FRONTEND=noninteractive

# hadolint ignore=DL3009
RUN --mount=type=cache,target=/var/lib/apt,sharing=locked,id=liquidsoap_2.2 \
    apt-get update

# renovate: datasource=repology depName=ubuntu_22_04/bash versioning=loose
ARG BASH_VERSION=5.1-6ubuntu1.1
# renovate: datasource=repology depName=ubuntu_22_04/util-linux versioning=loose
ARG UTIL_LINUX_VERSION=2.37.2-4ubuntu3.3

# hadolint ignore=DL3015
RUN --mount=type=cache,target=/var/lib/apt,sharing=locked,id=liquidsoap_2.2 \
    set -eux; \
      apt-get install -y --only-upgrade \
        bash="${BASH_VERSION}" \
        bsdutils="1:${UTIL_LINUX_VERSION}" \
        libblkid1="${UTIL_LINUX_VERSION}" \
        libmount1="${UTIL_LINUX_VERSION}" \
        libsmartcols1="${UTIL_LINUX_VERSION}" \
        libuuid1="${UTIL_LINUX_VERSION}" \
        mount="${UTIL_LINUX_VERSION}" \
        util-linux="${UTIL_LINUX_VERSION}" \
      ; \
      rm -f \
        /var/lib/dpkg/status-old \
        /var/cache/debconf/templates.dat-old \
        /var/cache/debconf/config.dat-old \
      ;

# renovate: datasource=repology depName=ubuntu_22_04/ca-certificates versioning=loose
ARG CA_CERTIFICATES_VERSION=20230311ubuntu0.22.04.1
# renovate: datasource=repology depName=ubuntu_22_04/gosu versioning=loose
ARG GOSU_VERSION=1.14-1
# renovate: datasource=repology depName=ubuntu_22_04/tzdata versioning=loose
ARG TZDATA_VERSION=2024a-0ubuntu0.22.04

RUN --mount=type=cache,target=/var/lib/apt,sharing=locked,id=liquidsoap_2.2 \
    --mount=type=cache,target=/packages,source=/packages,from=downloader \
    set -eux; \
      apt-get install -y --no-install-recommends \
        ca-certificates="${CA_CERTIFICATES_VERSION}" \
        gosu="${GOSU_VERSION}" \
        tzdata="${TZDATA_VERSION}" \
        "/packages/liquidsoap.deb" \
      ; \
      rm -f \
        /var/lib/dpkg/status-old \
        /var/cache/debconf/templates.dat-old \
        /var/cache/debconf/config.dat-old \
      ;

COPY rootfs /

ENTRYPOINT [ "entrypoint" ]
CMD [ "liquidsoap", "/var/liquidsoap/default.liq" ]
2.2-minimal/Dockerfile 
FROM alpine:3.19.1@sha256:c5b1261d6d3e43071626931fc004f70149baeba2c8ec672bd4f27761f8e1ad6b as downloader

# renovate: datasource=github-tags depName=savonet/liquidsoap-release-assets
ARG LIQUIDSOAP_VERSION=2.2.4

ARG LIQUIDSOAP_RELEASE="https://github.com/savonet/liquidsoap-release-assets/releases/download/v${LIQUIDSOAP_VERSION}"
ARG LIQUIDSOAP_DEB="liquidsoap-minimal_${LIQUIDSOAP_VERSION}-ubuntu-jammy-2_amd64.deb"
ARG LIQUIDSOAP_SHA512SUM="a99e2739d4cec26bd1cbe21d1cb32204e290d05fe449f770c7a90dc935d46ed39b337667f10996acc1e6bb37a47ca3bb26fc0561a7f3a04d157525e278321b3f"

# hadolint ignore=DL3047,DL4006
RUN set -eux; \
      mkdir /packages; \
      wget "${LIQUIDSOAP_RELEASE}/${LIQUIDSOAP_DEB}" -O /packages/liquidsoap.deb; \
      echo "${LIQUIDSOAP_SHA512SUM}  /packages/liquidsoap.deb" | sha512sum -c -;

FROM ubuntu:22.04@sha256:77906da86b60585ce12215807090eb327e7386c8fafb5402369e421f44eff17e

ENV TZ=UTC
ARG DEBIAN_FRONTEND=noninteractive

# hadolint ignore=DL3009
RUN --mount=type=cache,target=/var/lib/apt,sharing=locked,id=liquidsoap_2.2-minimal \
    apt-get update

# renovate: datasource=repology depName=ubuntu_22_04/bash versioning=loose
ARG BASH_VERSION=5.1-6ubuntu1.1
# renovate: datasource=repology depName=ubuntu_22_04/util-linux versioning=loose
ARG UTIL_LINUX_VERSION=2.37.2-4ubuntu3.3

# hadolint ignore=DL3015
RUN --mount=type=cache,target=/var/lib/apt,sharing=locked,id=liquidsoap_2.2-minimal \
    set -eux; \
      apt-get install -y --only-upgrade \
        bash="${BASH_VERSION}" \
        bsdutils="1:${UTIL_LINUX_VERSION}" \
        libblkid1="${UTIL_LINUX_VERSION}" \
        libmount1="${UTIL_LINUX_VERSION}" \
        libsmartcols1="${UTIL_LINUX_VERSION}" \
        libuuid1="${UTIL_LINUX_VERSION}" \
        mount="${UTIL_LINUX_VERSION}" \
        util-linux="${UTIL_LINUX_VERSION}" \
      ; \
      rm -f \
        /var/lib/dpkg/status-old \
        /var/cache/debconf/templates.dat-old \
        /var/cache/debconf/config.dat-old \
      ;

# renovate: datasource=repology depName=ubuntu_22_04/ca-certificates versioning=loose
ARG CA_CERTIFICATES_VERSION=20230311ubuntu0.22.04.1
# renovate: datasource=repology depName=ubuntu_22_04/gosu versioning=loose
ARG GOSU_VERSION=1.14-1
# renovate: datasource=repology depName=ubuntu_22_04/tzdata versioning=loose
ARG TZDATA_VERSION=2024a-0ubuntu0.22.04

RUN --mount=type=cache,target=/var/lib/apt,sharing=locked,id=liquidsoap_2.2-minimal \
    --mount=type=cache,target=/packages,source=/packages,from=downloader \
    set -eux; \
      apt-get install -y --no-install-recommends \
        ca-certificates="${CA_CERTIFICATES_VERSION}" \
        gosu="${GOSU_VERSION}" \
        tzdata="${TZDATA_VERSION}" \
        "/packages/liquidsoap.deb" \
      ; \
      rm -f \
        /var/lib/dpkg/status-old \
        /var/cache/debconf/templates.dat-old \
        /var/cache/debconf/config.dat-old \
      ;

COPY rootfs /

ENTRYPOINT [ "entrypoint" ]
CMD [ "liquidsoap", "/var/liquidsoap/default.liq" ]

I build them both with the same command in parallel docker build --network host -t "${CONTAINER_BUILD_IMAGE}:${CONTAINER_TAG}" "./containers/${CONTAINER_TAG}", just using different environment variables.

P.S. I haven't figured out how to use AND invalidate cache using files or external registry, so I don't use cache at all when building in CI.

from moby.

thaJeztah avatar thaJeztah commented on September 20, 2024

it solves the problem of building multiple images that look the same on the local machine at the same time. In CI, I build each image in a separate namespace so they don't overlap.

I build them both with the same command in parallel (...) just using different environment variables.

The parts that depend on env-vars / build-args should still be invalidated if those parameters are different. I'm curious though why the common steps should be executed again in your situation? If the problem is that the apt-get update is not executed again when changing these options, then it's a better option to combine that with the step that performs the install (i.e., if you want apt-get update to be updated when you update the versions of packages to install).

Can you describe for which steps in your example you expected the cache to be invalidated, but that not happening? (And how they differ between the different Dockerfiles (or same Dockerfile, but different parameters)?

from moby.

vitoyucepi avatar vitoyucepi commented on September 20, 2024

Can you describe for which steps in your example you expected the cache to be invalidated

I didn't design my dockerfiles for cache invalidation. On localhost I just remove the build cache. In CI I don't use build cache at all.
And now I'm in the process of refactoring my scripts.

from moby.

cpuguy83 avatar cpuguy83 commented on September 20, 2024

Why is it using "type=cache" for mounting the content of "downloader"?
Shouldn't that be "type=bind" (also the default)?

from moby.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.