Since a recent update our conda package building pipe line broke.

Specifically, it fails on:

➜  ~ docker run -it condaforge/linux-anvil
(root) [conda@9e93de846f43 ~]$ yum install -y devtoolset-2-gcc-gfortran
Loaded plugins: fastestmirror, ovl
ovl: Error while doing RPMdb copy-up:
[Errno 13] Permission denied: '/var/lib/rpm/Conflictname'
You need to be root to perform this command.

FYI I was trying to run linux-anvil-comp7 this morning from a WSL2 (ubuntu) environment, and had to employ the workaround in microsoft/WSL#4694 (comment) to avoid an immediate docker exit with code 139.

Not sure if there's much that should be done about this, hopefully this note helps others with the same problem.

I get the following error when I try to run yum update using the latest docker image:

$ docker run --rm condaforge/linux-anvil:latest yum update
bash: cannot set terminal process group (-1): Inappropriate ioctl for device
bash: no job control in this shell
Loaded plugins: fastestmirror, ovl
ovl: Error while doing RPMdb copy-up:
[Errno 13] Permission denied: '/var/lib/rpm/Sha1header'
You need to be root to perform this command.

The image for circle-ci (centos I believe) has no unzip command available. I think it would be useful to add it. E.g. github downloads are zip by default (and it is not that easy to find how to get a tarball).

It seems many of our other maintenance repos start with a conda-forge- prefix. Wondering whether this one should follow that trend.

Issue: It appears that there's no system wide Perl installation in condaforge/linux-anvil-comp7. This is not really a big deal since Perl is installed in the base environment:

[conda@7cf862609754 ~]$ which perl
/opt/conda/bin/perl

However, I found out that some applications look for perl in specific locations, namely /usr/bin/perl. For e.g: The cuda-toolkit installer expects to find Perl in /usr/bin/perl: conda-forge/cudatoolkit-dev-feedstock#11

/opt/conda/pkgs/cudatoolkit-dev/cuda_10.0.130_410.48_linux: ./cuda-installer.pl: /usr/bin/perl: bad interpreter: No such file or directory

I am wondering whether there exists a simple fix here or whether this would require having a system wide Perl installation as part of the docker image?

This build is pretty hefty so we should move it to azure.

In a recent test I found that the glibc version in CF's CUDA docker images is too old (2.12), and it is unable to support new CUDA libraries such as cuTENSOR which needs 2.14+ (conda-forge/cupy-feedstock#57).

Quoting @jjhelmus from conda-forge/cupy-feedstock#57 (comment):

The CUDA Toolkit 10.0 release notes indicated that support for RHEL 6.x was deprecated and may be dropped in a future release. I would not be surprised if cuTENSOR did not support RHEL 6.x.

RHEL 6 / CentOS 6 use glibc 2.12, the later is the base for the conda-forge/linux-anvil-cuda:10.0 image.

Would it make sense to not only include mamba (And where necessary mambaforge) in the image but also build it with that?

Happy to create the PR but wanted to get some opinions first.

There are currently at least two proposals for build images:

conda-forge/staged-recipes#97
and
conda-forge/staged-recipes#97 (comment)

Both are based on CentOS 5.11. The former is a stripped-down version of the manylinux docker image that includes Miniconda. The latter builds GCC 5.2 instead of packaging the RedHat devtools (GCC 4.8) that manylinux (and many others) depend on.

This issue's discussion should cover:

• what tools to include in a standard image, and which to leave to conda packages (explicitly declared in recipes)
• what version of GCC (and where to get it)?

Is make necessary in the docker image?

It seems we can always add it to the conda dependencies right?

So centos bundles binutils, and the whole OS depends on them.

If you want to make sure you don't use them, I would recommend adding a one liner that just removes them.

rm -f /usr/bin/{ar,addr2line,as,c++filt,dwp,elfedit,gprof,ld,ld.bfd,ld.gold,nm,objcopy,objdump,ranlib,readlef,size,strings,strip}

This is going to make many builds frustrating when they fail on a missing ar when the build is almost over....

https://github.com/Archiconda/conda-smithy/blob/aarch64/conda_smithy/templates/shippable.yml.tmpl#L44

Several of my builds have started failing with the most recent condaforge/linux-anvil docker image (for example, see here).

  File "/opt/conda/bin/conda-build", line 11, in 
    sys.exit(main())
  File "/opt/conda/lib/python3.6/site-packages/conda_build/cli/main_build.py", line 413, in main
    execute(sys.argv[1:])
  File "/opt/conda/lib/python3.6/site-packages/conda_build/cli/main_build.py", line 404, in execute
    verify=args.verify)
  File "/opt/conda/lib/python3.6/site-packages/conda_build/api.py", line 188, in build
    need_source_download=need_source_download, config=config, variants=variants)
  File "/opt/conda/lib/python3.6/site-packages/conda_build/build.py", line 1913, in build_tree
    notest=notest,
  File "/opt/conda/lib/python3.6/site-packages/conda_build/build.py", line 1053, in build
    is_conda=m.name() == 'conda')
  File "/opt/conda/lib/python3.6/site-packages/conda_build/environ.py", line 812, in create_env
    execute_actions(actions, index, verbose=config.debug)
  File "/opt/conda/lib/python3.6/site-packages/conda/common/io.py", line 441, in decorated
    return f(*args, **kwds)
  File "/opt/conda/lib/python3.6/site-packages/conda/plan.py", line 392, in execute_actions
    execute_instructions(plan, index, verbose)
  File "/opt/conda/lib/python3.6/site-packages/conda/plan.py", line 599, in execute_instructions
    cmd(state, arg)
  File "/opt/conda/lib/python3.6/site-packages/conda/instructions.py", line 64, in PROGRESSIVEFETCHEXTRACT_CMD
    progressive_fetch_extract.execute()
  File "/opt/conda/lib/python3.6/site-packages/conda/core/package_cache.py", line 590, in execute
    raise CondaMultiError(exceptions)
conda.CondaMultiError: [Errno 21] Is a directory: '/opt/conda/pkgs/setuptools-38.4.0-py36_0'

I am using the most recent version of conda (as obtained using conda update -n root conda conda-build), and I think the packages inside the /opt/conda/pkgs folder are somehow too old for conda to handle them gracefully. Running rm -rf /opt/conda/pkgs/* before conda build . seems to fix the issue.

Maybe rm -rf /opt/conda/pkgs/* should be included in the Dockerfile? This would have an additional advantage of making the docker image smaller.

Not sure if I have the right repo. Sorry still figuring this stuff out. :)

So, I have a recipe that uses Qt, but it is missing some dependencies like libXext.so.6. Would be nice to add these to the image.

cc @jjhelmus, I think I saw one of your feedstocks (don't remember which one) adds this stuff explicitly. Is that the way we should do this or can we add them to the docker image?

I am not really sure how to even go about figuring this out, but I am not sure why a script in this build cannot see the user and mount namespaces enabled inside the centos containers. The charliecloud script automatically scans for these two namespaces to determine what to build (ch-image, etc.)

Running containers
~~~~~~~~~~~~~~~~~~

  ch-run(1): no
    user+mount namespaces ... no

conda-forge/staged-recipes#17238

This is weird, because I am usually accustomed to checking namespaces via lsns and this is what you'd get inside the centos containers:

~$ docker run -it quay.io/condaforge/linux-anvil-comp7      
[conda@34b6d87eaaa6 ~]$ lsns
bash: lsns: command not found
[conda@34b6d87eaaa6 ~]$ 

~$ docker run -it condaforge/mambaforge
(base) root@f9d1f441da16:/# lsns
        NS TYPE   NPROCS PID USER COMMAND
4026531837 user        3   1 root tini -- /bin/bash
4026532372 mnt         3   1 root tini -- /bin/bash
4026532373 uts         3   1 root tini -- /bin/bash
4026532374 ipc         3   1 root tini -- /bin/bash
4026532375 pid         3   1 root tini -- /bin/bash
4026532377 net         3   1 root tini -- /bin/bash
4026532453 cgroup      3   1 root tini -- /bin/bash

$ docker run -it quay.io/condaforge/linux-anvil-cos7-x86_64
[conda@7ca974e6e3a7 ~]$ lsns
        NS TYPE  NPROCS PID USER  COMMAND
4026531837 user       2   7 conda /usr/bin/qemu-x86_64 /bin/bash
4026532287 mnt        2   7 conda /usr/bin/qemu-x86_64 /bin/bash
4026532288 uts        2   7 conda /usr/bin/qemu-x86_64 /bin/bash
4026532289 ipc        2   7 conda /usr/bin/qemu-x86_64 /bin/bash
4026532290 pid        2   7 conda /usr/bin/qemu-x86_64 /bin/bash
4026532292 net        2   7 conda /usr/bin/qemu-x86_64 /bin/bash

Anyway, any experience with this or any immediate thoughts?

• I read the conda-forge documentation and could not find the solution for my problem there.

Issue:

$ conda list

$ conda info

As our current docker image is being moved to community maintenance ( #5 ), we should probably update the MAINTAINER tag to reflect this. Unfortunately, I don't think the MAINTAINER tag responds well to multiple authors and maybe that is not the way for us to go anyways. I propose the following adjustment. Feedback welcome.

MAINTAINER conda-forge <[email protected]>

Getting the following error on travis:

Status: Downloaded newer image for condaforge/linux-anvil-ppc64le:latest
standard_init_linux.go:190: exec user process caused "exec format error"

It seems that this is because the image is lacking bash? containers/buildah#475 (comment)

It is sometimes convenient to use the condaforge/linux-anvil container locally (i.e. without a feedstock) for development and testing. Of course, one occasionally needs to use sudo within the container.

Confusingly, devtoolset-2 installs its own version of sudo (or a wrapper of it?):

$ which sudo
/opt/rh/devtoolset-2/root/usr/bin/sudo

That version of sudo is first on the container's PATH, but unfortunately it seems to be broken, so commands like the following will fail:

$ sudo yum install libX11-devel

(It rejects a blank password.)

The correct alternative is to use the system's original version: /usr/bin/sudo

$ /usr/bin/sudo yum install libX11-devel

But most users probably wouldn't guess to try that. Is there some way we can make plain sudo work? Possibilities:

1. Delete /opt/rh/devtoolset-2/root/usr/bin/sudo
2. Move /usr/bin ahead of /opt/rh/devtoolset-2/root/usr/bin on the path
3. cd /opt/conda/bin && ln -s /usr/bin/sudo (since /opt/conda/bin is already first on the PATH)

I guess I prefer option 1. Thoughts?

now that we also have mambaforge, it would be great to add a docker image to the docker hub.

The Dockerfile is the same of miniforge, replacing the name with mamabaforge.

Please consider renaming the jnlp-slave Docker config and the corresponding image name(s). Some background: https://bugs.python.org/issue34605. Thank you.

As setuptools has now made it into the conda-build requirements ( conda-forge/conda-build-feedstock#101 ), it would be nice to drop it from this install line. However we will likely need to wait for a few conda-build releases until things have had a chance to settled down a bit more.

We need to tweak the image to make sure USER is defined. On a normal Linux system this would be defined, but that is not the case here. However, this is very easy to change. This is coming up because of this proposed recipe ( conda-forge/staged-recipes#477 ).

Sounds like we don't want to call it obvious-ci. So, let's figure out what we want to call it.

It appears that #181 did not fix the problem, see conda-forge/cutensor-feedstock#12 (comment).

It would be good to have test images that represent user systems. Ideally, more than one platform could be tested. This is all subject to time constraints on how much time tests can take.

Proposed platforms:

• CentOS 5
• Some current RHEL-based release
• Some old Debian
• Some current Ubuntu release (especially with the new GCC 5, to test how packages are affected by that)

On my local machine, the chown on the conda folder (here) takes about 8 min to complete. Probably this has not been an issue before, but only occurs now after the meltdown patches? I guess this might also lead to considerable slowdowns in CI. I wonder if there is a way to avoid the chown?

We need a CentOS 7 image for building CUDA, especially CUDA 11.0. I am sure it's on the roadmap, just opening an issue here for cross referencing from other places 😁

Could this include the arch string in the miniforge download URL?

https://github.com/conda-forge/miniforge/releases

• Miniforge3-${MINIFORGE_VERSION}-Linux-x86_64.sh
• Miniforge3-${MINIFORGE_VERSION}-Linux-aarch64.sh
• Miniforge3-${MINIFORGE_VERSION}-Linux-ppe64le.sh

How to make the Dockerfile pass the arch through to be templated into the miniforge release URL?

• In docker experimental mode, the --platform flag works
  • it sets DEB_TARGET_GNU_CPU=aarch64
• In lieu of waiting for things to emerge from experimental mode in the moby project, specifying the CPU arch as a build ARG or ENV variable (TARGET_CPU_ARCH=x86_64 | aarch64 | ppe64le may be the most portable way to support multiple architectures (and any container builder) with the miniforge docker container.

...

• If multi-OS is also a goal, the ansible molecule dockerfile shows one way to execute different package commands depending upon the existence of various package managers:
  https://github.com/ansible-community/molecule/blob/master/molecule/data/Dockerfile.j2#L17-L22

The license file added here was MIT. Though typically we use BSD 3-Clause with source code. Are we happy with MIT license here or do we wish to switch it to BSD 3-Clause?

cc @conda-forge/core

Hi conda-forge team,

I was wondering that since you're using centos:6 as the basis for the docker container - wouldn't this cause problems with the software meant to be run and built on Ubuntu?

Excuse my ignorance, since I'm new to a lot of stuff here but please do answer this question, I'm curios.

Currently the ppc64le images lack cudatoolkit as noted here. Ideally we would like to cache the cudatoolkit packages as is done in the x86_64 images. In order to do this we would need to build out cudatoolkit packages with ppc64le support, which amounts to making a couple PRs ( AnacondaRecipes/cudatoolkit-feedstock#4 ). Once that is done and the packages built, it should be straightforward to add these lines to the Docker image.

Given we have migrated the stack to the newer conda-forge/linux-anvil-comp7 (and derivatives) and we are not maintaining the legacy conda-forge/linux-anvil image, should we go ahead and remove it from this repo?

Right now it doesn't seem there is a GitHub team for this repo. Seems like it would be easier for org maintenance and communication if there was one. Guessing docker-images would be the team name. Though we may need to update it if we rename the repo ( #25 ).

We have a Docker image on Docker Hub called conda-forge-x86_64. It hasn't seen any pushes in 5mos, it has only a handful of pulls (17), was manually pushed, and AFAICT is unused. Few questions.

1. What is it?
2. Do we need it?
3. If we don't need it, can it be removed?

Currently the images just have a "latest" tag, and over on bioconda we recently had builds failing due to changes in the linux-anvil image we weren't aware of. Pinning to a tagged image would help this sort of thing in the future. @jakirkham suggested git sha as tags, which for our purposes would be great

xref bioconda/bioconda-recipes#3080 (comment)

Since we include some components we may want to cache inside the images -- it might be a good idea to build the docker containers daily instead of ad-hoc. This should be able to reduce the time that linux builds take to build by avoiding needing to solve and download a subset of the packages.

https://hub.docker.com/r/condaforge/linux-anvil/builds/bo2anargh9vx5zchqrdwmrr/

Removing intermediate container 67f92b03cba2
Step 4 : RUN yum update -y &&     yum install -y                    bzip2                    gcc-c++                    git                    make                    patch                    tar                    which                    libXext-devel                    libXrender-devel                    libSM-devel                    libX11-devel                    mesa-libGL-devel &&     yum clean all
 ---> Running in 467dbb0d7283
Loaded plugins: fastestmirror
Setting up Update Process
�[91mError: Cannot retrieve repository metadata (repomd.xml) for repository: extras. Please verify its path and try again
�[0mCould not retrieve mirrorlist http://mirrorlist.centos.org/?release=6&arch=x86_64&repo=extras&infra=stock error was
14: PYCURL ERROR 22 - "The requested URL returned error: 500 Internal Server Error"
Removing intermediate container 467dbb0d7283
The command '/bin/sh -c yum update -y &&     yum install -y                    bzip2                    gcc-c++                    git                    make                    patch                    tar                    which                    libXext-devel                    libXrender-devel                    libSM-devel                    libX11-devel                    mesa-libGL-devel &&     yum clean all' returned a non-zero code: 1

Hi!

I'd like to use the linux-anvil image to debug, why the a pkg I am working on, does not build correctly. But

1. I cannot run the docker image grom docker-hub
2. I cannot build the docker image myself using the Dockerfile from this repo

1. when i run docker run -it condaforge/linux-anvil /bin/bash I get a return code 139 - that's it. Other images run without problems. That's why I tried to build it myself, but
2. when building I can this

Step 1/15 : FROM centos:6
 ---> 70b5d81549ec
Step 2/15 : MAINTAINER conda-forge <[email protected]>
 ---> Using cache
 ---> c90459fe5325
Step 3/15 : ENV LANG en_US.UTF-8
 ---> Using cache
 ---> 20d6b327c60c
Step 4/15 : ADD http://tycho.usno.navy.mil/timer.html /opt/docker/etc/timestamp
Downloading [==================================================>]     561B/561B
 ---> a9b19bb103bc
Step 5/15 : RUN rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6
 ---> Running in d5ba4e28bb3b
The command '/bin/sh -c rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6' returned a non-zero code: 139```

and again here is returncode 139. My impresion is, that this comes from within the centos:6 image. 

Does anybody have a solution for this?

thx in advance

The layer created here has ~1.4GB size:

Broken in #80

cc @sodre

+ /usr/bin/sudo -n yum install -y mesa-libGL mesa-dri-drivers libselinux libXdamage libXxf86vm libXext libXfixes libXinerama libXrandr libXcursor libXcomposite libX11 libXi
Loaded plugins: fastestmirror, ovl
Setting up Install Process
Determining fastest mirrors
Error: Cannot find a valid baseurl for repo: base
YumRepo Error: All mirror URLs are not using ftp, http[s] or file.
 Eg. Invalid release/repo/arch combination/
removing mirrorlist with no valid mirrors: /var/cache/yum/x86_64/6/base/mirrorlist.txt

See build pipelines in conda-forge/hugin-feedstock#7

There may be good reason for this, I noticed that the linux-anvil-aarch64 and linux-anvil-ppc64le Dockerfiles don't install make but the linux-anvil-comp7 Does.

Is that by design?

Seems this repo is using GitHub issue/PR templates designed for feedstocks, which doesn't make much sense here. Would be nice to disable them here short term and possibly update them later.

Ass suggested by @isuruf, it would be nice to have a mechanism that installs a custom conda installer instead of the predefined ones:

That would be useful to test the Miniforge installers created by https://github.com/conda-forge/miniforge. See conda-forge/miniforge#2 (comment) for details.

Issue:
Conda 4.8.4 is released, containing some bug fixes. Can we update the images to use this version?


Environment (conda list):

$ conda list

$ conda info

Appears they are running into an issue getting Conda 4.8.5. Not sure why that version is being used

Collecting package metadata (current_repodata.json): ...working... done
1386
Solving environment: ...working... failed with initial frozen solve. Retrying with flexible solve.
1387
Solving environment: ...working... failed with repodata from current_repodata.json, will retry with next repodata source.
1388

1389
ResolvePackageNotFound: 
1390
  - conda==4.8.5

https://github.com/conda-forge/docker-images/runs/3648769402

Am interested in seeing how we can standardize the images here a bit and consolidate maintenance effort. In particular am curious how we can make a multi-arch image. There's a good blogpost on this topic. It would be good to determine what the current issues prompting the different images we have are and how we address those issues in a general way that benefits all of the images here.

https://travis-ci.org/conda-forge/docker-images/jobs/511844961

cc @ocefpaf

More and more, I'm finding that conda is starting to fail to find solutions to environments. I'm not sure if it is because I have channels other than conda-forge enabled

is it time to start including mamba in the anvil images so as to speed up build time on systems?

The DockerHub builds are having some sort of checksum error when installing some things. As this is not happening with CircleCI and I am unable to reproduce it locally, it seems like a DockerHub issue. Added the following upstream issue ( docker/hub-feedback#675 ). Let's hold on new changes for a bit.

If DockerHub remains unstable in these sorts of ways (sadly not the first issue I have encountered with automated builds on DockerHub), we could try quay.io or we could just push images from CircleCI. If we try the former, we can mirror images on DockerHub to avoid having to switch a bunch of links.