provide a way to build RHEL images on Fedora box about container-common-scripts HOT 30 CLOSED

Per @hhorak report, it is possible to (optionally) provide -v /host.repo:/etc/yum.repos.d/host.repo option for docker build . command.

Unfortunately this does not work as docker build makes sure every file is mounted as read-only and the first thing we are doing in every image is disabling all repositories, making the build fail.

from container-common-scripts.

the first thing we are doing in every image is disabling all repositories

Do we still have to do this?

from container-common-scripts.

Do we still have to do this?

I think so. Afaik this is present so that we do not install any packages from other repositories that might be enabled on the host.

from container-common-scripts.

The yum-config-manager --enable ... commands would have to be removed as well, since they also try to save the .repo file changes.

from container-common-scripts.

Looking into this again since we are now able to use prepare-yum-repositories script in images built from s2i-core.

@hhorak Is there an easy way to set SKIP_REPOS_ENABLE on build? I have only found docker build --build-args ... so far but it seems that an additional ARG instruction would need to be added to every Dockerfile to enable that functionality...

from container-common-scripts.

The other way to use prepare-yum-repositories with SKIP_REPOS_ENABLE set is to modify the actual Dockerfile before the build:

https://github.com/pkubatrh/container-common-scripts/blob/custom_repo/build.sh#L74

This makes sure that the custom repo file is mounted into the build environment (as read-only) and not touched (no enabling/disabling) for the whole duration of the build.

from container-common-scripts.

@pkubatrh So, the ARG is not inherited?

from container-common-scripts.

@pkubatrh So, the ARG is not inherited?

It does not seem to be. Which is imo the expected behaviour as the ARG instruction (and the --build-args argument) is tied to the build of the Dockerfile.

from container-common-scripts.

Is there an easy way to set SKIP_REPOS_ENABLE on build? I have only found docker build --build-args ... so far but it seems that an additional ARG instruction would need to be added to every Dockerfile to enable that functionality...

ONBUILD could be used in centos/s2i-core-centos7 and in centos/s2i-base-centos7 - https://docs.docker.com/engine/reference/builder/#onbuild . (And potentially in other other images which start to serve as a base image...)

from container-common-scripts.

ONBUILD could be used in centos/s2i-core-centos7 and in centos/s2i-base-centos7

You are right, that should do the trick. Thanks for the idea! Will prepare a PoC

from container-common-scripts.

Hm, nice idea :)
So, what else we need in order to build the RHEL images on non-RHEL machine? Maybe one more variable that would be set in case a user wants to add a remote repo URL that would be used instead?

from container-common-scripts.

Maybe one more variable that would be set in case a user wants to add a remote repo URL that would be used instead?

I was thinking a Makefile argument that would provide the path to the repository file (or a directory containing the repository files?) which would then be read-only mounted into the container at build time with SKIP_REPOS_ENABLE set so that no modifications are tried to be made to the mounted files.

from container-common-scripts.

pkubatrh@7beee13

from container-common-scripts.

Although I am not sure if that would be of any use for users running in Openshift. Having another variable for remote repository information would work for them.

from container-common-scripts.

One annoying thing with using the ARG instruction is that when it is used it introduces cache misses for every instruction after it that could be using that build-arg. With ONBUILD ARG ... injecting the instruction directly after FROM this means no cache for the whole Dockerfile...

from container-common-scripts.

One annoying thing with using the ARG instruction is that when it is used it introduces cache misses for every instruction after it that could be using that build-arg. With ONBUILD ARG ... injecting the instruction directly after FROM this means no cache for the whole Dockerfile...

Agree.

Another way could be to forgot about ARG instruction and use only volumes during docker build. Mount own repositories into /etc/yum.repos.d/ and in prepare-yum-repositories (which is called in all images) check that nothing else than /etc/yum.repos.d/redhat.repo is present. If some different repo file is there (user has provided it) we know that disabling of repositories should be skipped.

from container-common-scripts.

Agreed with the last comment, plus one option is that we could mount the repofile directory
to some dedicated path (say /etc/yum.repos.containerbuild/) and prepare-yum-repositories could copy the files to /etc/yum.repos.d ... that works-around the read-only issues.

The common.mk should (optionally) accept variable with path to the hots's directory with repofiles.

from container-common-scripts.

@praiskup The read-only bind is a non-issue as long as we can determine that the user wants to use custom repositories so that we can remove all the repository disabling/enabling in prepare-yum-repositories. This can also be a feature - there is at least one bugzilla I know of that would welcome a way to disable repository disabling on build.

Mount own repositories into /etc/yum.repos.d/ and in prepare-yum-repositories (which is called in all images) check that nothing else than /etc/yum.repos.d/redhat.repo is present. If some different repo file is there (user has provided it) we know that disabling of repositories should be skipped.

This is a good idea, thanks! However, I would like to make sure it works with remote content as well.

from container-common-scripts.

@pkubatrh wrote:

However, I would like to make sure it works with remote content as well.

Can you elaborate this?

from container-common-scripts.

Can you elaborate this?

Instead of providing a local path to the repo files which would then be mounted into the image at build-time, a remote url from where the file need to be fetched will be passed to the image on build (via some environment variable?).

My first thought was Openshift use-cases, but it might be possible to set up the build nodes in such a way that they always have the repo files stored somewhere accessible locally during each build but I am not that knowledgeable when it comes to Openshift to know for certain.

So not sure if this is really needed, but it would be nice to have.

from container-common-scripts.

I'm not sure whether we can think about openshift usecases; do you mean s2i? There you don't
install packages (root needed). And if prepare-yum-repositories bakes something into
/etc/yum.repos.d it will be persistent anyways ...

from container-common-scripts.

And if prepare-yum-repositories bakes something into
/etc/yum.repos.d it will be persistent anyways ...

Do we want this to happend by default?

from container-common-scripts.

I'm not sure whether we can think about openshift usecases; do you mean s2i?

Not s2i, just plain old Docker builds using buildconfigs (what openshift calls docker strategy).

from container-common-scripts.

The use case I am talking about is basically this: https://bugzilla.redhat.com/show_bug.cgi?id=1514989

from container-common-scripts.

There are definitely more scenarios that we may address here and I start to be lost a bit. Trying to summarize them all:

1. Subscribed RHEL host: build should ensure correct repos are enabled in redhat.repo (default behaviour)
2. Subscribed RHEL host: use exactly repos that are enabled on host
3. any host: use repo file from URL
4. any host: use local repo file
   Anything I missed?

And there are generally these ways to change building process:
a) ARG instruction
b) mounting repo file read-only
c) having a file in image sources
Again, anything missed?

I'm thinking about one more thing that was probably not mentioned so far -- we may have an empty directory root/etc/yum.repos.d in all sources, and put the content of this directory to the image before RPMs installation. Putting files into this directory would have the same effect as mounting them, but it would also work fine in OpenShift.

from container-common-scripts.

One annoying thing with using the ARG instruction is that when it is used it introduces cache misses for every instruction after it that could be using that build-arg.

Does it really happen only when ARG is used, i.e. defined during docker build? If so, I'd take this as a price for using this feature, if other cases are not affected.

from container-common-scripts.

Does it really happen only when ARG is used, i.e. defined during docker build? If so, I'd take this as a price for using this feature, if other cases are not affected.

Actually, I need to correct myself here - it seems it is not ARG that is causing the cache misses, repo file mounting is the real culprit. Which I guess is the correct behaviour since mounting does not actually introduce any changes into the image layers by itself (the files are present only for the duration of the build).

from container-common-scripts.

IMO, this is test-only feature, I wouldn't re-purpose this to some production-ready thing.
Simply, if one want's to build RHEL based image and test that testsuite runs, OK, but
I would rather stick with the fact that if the images are to be really used in production,
(e.g. in openshift) then one should either get the subscription or use the official images.

Otherwise, it seems like everything was said.... it would be nice if the feature was an easy
opt-in. I would let other decision to the patch submitter.

from container-common-scripts.

Would not call this a test-only feature - more like a development-only feature. Which is why I think other users would benefit from it.

In any case I will try to work something out and follow up with a PR.

from container-common-scripts.

I think this is done already so closing

from container-common-scripts.