conda-forge / docker-images Goto Github PK
View Code? Open in Web Editor NEWRepository to host the Docker images files used in conda-forge
Home Page: https://hub.docker.com/u/condaforge/
License: BSD 3-Clause "New" or "Revised" License
Repository to host the Docker images files used in conda-forge
Home Page: https://hub.docker.com/u/condaforge/
License: BSD 3-Clause "New" or "Revised" License
Currently the ppc64le images lack cudatoolkit
as noted here. Ideally we would like to cache the cudatoolkit
packages as is done in the x86_64 images. In order to do this we would need to build out cudatoolkit
packages with ppc64le
support, which amounts to making a couple PRs ( AnacondaRecipes/cudatoolkit-feedstock#4 ). Once that is done and the packages built, it should be straightforward to add these lines to the Docker image.
More and more, I'm finding that conda is starting to fail to find solutions to environments. I'm not sure if it is because I have channels other than conda-forge enabled
is it time to start including mamba
in the anvil images so as to speed up build time on systems?
Getting the following error on travis:
Status: Downloaded newer image for condaforge/linux-anvil-ppc64le:latest
standard_init_linux.go:190: exec user process caused "exec format error"
It seems that this is because the image is lacking bash? containers/buildah#475 (comment)
Hi!
I'd like to use the linux-anvil image to debug, why the a pkg I am working on, does not build correctly. But
docker run -it condaforge/linux-anvil /bin/bash
I get a return code 139 - that's it. Other images run without problems. That's why I tried to build it myself, butStep 1/15 : FROM centos:6
---> 70b5d81549ec
Step 2/15 : MAINTAINER conda-forge <[email protected]>
---> Using cache
---> c90459fe5325
Step 3/15 : ENV LANG en_US.UTF-8
---> Using cache
---> 20d6b327c60c
Step 4/15 : ADD http://tycho.usno.navy.mil/timer.html /opt/docker/etc/timestamp
Downloading [==================================================>] 561B/561B
---> a9b19bb103bc
Step 5/15 : RUN rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6
---> Running in d5ba4e28bb3b
The command '/bin/sh -c rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6' returned a non-zero code: 139```
and again here is returncode 139. My impresion is, that this comes from within the centos:6 image.
Does anybody have a solution for this?
thx in advance
Issue:
Conda 4.8.4 is released, containing some bug fixes. Can we update the images to use this version?
Environment (conda list
):
$ conda list
conda
and system ( conda info
):
$ conda info
FYI I was trying to run linux-anvil-comp7
this morning from a WSL2 (ubuntu) environment, and had to employ the workaround in microsoft/WSL#4694 (comment) to avoid an immediate docker exit with code 139.
Not sure if there's much that should be done about this, hopefully this note helps others with the same problem.
Ass suggested by @isuruf, it would be nice to have a mechanism that installs a custom conda installer instead of the predefined ones:
docker-images/scripts/run_commands
Line 7 in eb117c9
That would be useful to test the Miniforge installers created by https://github.com/conda-forge/miniforge. See conda-forge/miniforge#2 (comment) for details.
It would be good to have test images that represent user systems. Ideally, more than one platform could be tested. This is all subject to time constraints on how much time tests can take.
Proposed platforms:
Given we have migrated the stack to the newer conda-forge/linux-anvil-comp7
(and derivatives) and we are not maintaining the legacy conda-forge/linux-anvil
image, should we go ahead and remove it from this repo?
It is sometimes convenient to use the condaforge/linux-anvil
container locally (i.e. without a feedstock) for development and testing. Of course, one occasionally needs to use sudo
within the container.
Confusingly, devtoolset-2
installs its own version of sudo
(or a wrapper of it?):
$ which sudo
/opt/rh/devtoolset-2/root/usr/bin/sudo
That version of sudo
is first on the container's PATH
, but unfortunately it seems to be broken, so commands like the following will fail:
$ sudo yum install libX11-devel
(It rejects a blank password.)
The correct alternative is to use the system's original version: /usr/bin/sudo
$ /usr/bin/sudo yum install libX11-devel
But most users probably wouldn't guess to try that. Is there some way we can make plain sudo
work? Possibilities:
/opt/rh/devtoolset-2/root/usr/bin/sudo
/usr/bin
ahead of /opt/rh/devtoolset-2/root/usr/bin
on the pathcd /opt/conda/bin && ln -s /usr/bin/sudo
(since /opt/conda/bin
is already first on the PATH
)I guess I prefer option 1. Thoughts?
Sounds like we don't want to call it obvious-ci. So, let's figure out what we want to call it.
Could this include the arch string in the miniforge download URL?
docker-images/miniforge3/Dockerfile
Line 25 in 8ae623a
https://github.com/conda-forge/miniforge/releases
Miniforge3-${MINIFORGE_VERSION}-Linux-x86_64.sh
Miniforge3-${MINIFORGE_VERSION}-Linux-aarch64.sh
Miniforge3-${MINIFORGE_VERSION}-Linux-ppe64le.sh
How to make the Dockerfile pass the arch through to be templated into the miniforge release URL?
--platform
flag works
DEB_TARGET_GNU_CPU=aarch64
ARG
or ENV
variable (TARGET_CPU_ARCH=x86_64 | aarch64 | ppe64le
may be the most portable way to support multiple architectures (and any container builder) with the miniforge docker container....
I am not really sure how to even go about figuring this out, but I am not sure why a script in this build cannot see the user and mount namespaces enabled inside the centos containers. The charliecloud script automatically scans for these two namespaces to determine what to build (ch-image, etc.)
Running containers
~~~~~~~~~~~~~~~~~~
ch-run(1): no
user+mount namespaces ... no
conda-forge/staged-recipes#17238
This is weird, because I am usually accustomed to checking namespaces via lsns
and this is what you'd get inside the centos containers:
~$ docker run -it quay.io/condaforge/linux-anvil-comp7
[conda@34b6d87eaaa6 ~]$ lsns
bash: lsns: command not found
[conda@34b6d87eaaa6 ~]$
~$ docker run -it condaforge/mambaforge
(base) root@f9d1f441da16:/# lsns
NS TYPE NPROCS PID USER COMMAND
4026531837 user 3 1 root tini -- /bin/bash
4026532372 mnt 3 1 root tini -- /bin/bash
4026532373 uts 3 1 root tini -- /bin/bash
4026532374 ipc 3 1 root tini -- /bin/bash
4026532375 pid 3 1 root tini -- /bin/bash
4026532377 net 3 1 root tini -- /bin/bash
4026532453 cgroup 3 1 root tini -- /bin/bash
$ docker run -it quay.io/condaforge/linux-anvil-cos7-x86_64
[conda@7ca974e6e3a7 ~]$ lsns
NS TYPE NPROCS PID USER COMMAND
4026531837 user 2 7 conda /usr/bin/qemu-x86_64 /bin/bash
4026532287 mnt 2 7 conda /usr/bin/qemu-x86_64 /bin/bash
4026532288 uts 2 7 conda /usr/bin/qemu-x86_64 /bin/bash
4026532289 ipc 2 7 conda /usr/bin/qemu-x86_64 /bin/bash
4026532290 pid 2 7 conda /usr/bin/qemu-x86_64 /bin/bash
4026532292 net 2 7 conda /usr/bin/qemu-x86_64 /bin/bash
Anyway, any experience with this or any immediate thoughts?
Issue:
conda list
):
$ conda list
conda
and system ( conda info
):
$ conda info
Not sure if I have the right repo. Sorry still figuring this stuff out. :)
So, I have a recipe that uses Qt, but it is missing some dependencies like libXext.so.6
. Would be nice to add these to the image.
cc @jjhelmus, I think I saw one of your feedstocks (don't remember which one) adds this stuff explicitly. Is that the way we should do this or can we add them to the docker image?
We need to tweak the image to make sure USER
is defined. On a normal Linux system this would be defined, but that is not the case here. However, this is very easy to change. This is coming up because of this proposed recipe ( conda-forge/staged-recipes#477 ).
Hi conda-forge team,
I was wondering that since you're using centos:6
as the basis for the docker container - wouldn't this cause problems with the software meant to be run and built on Ubuntu?
Excuse my ignorance, since I'm new to a lot of stuff here but please do answer this question, I'm curios.
On my local machine, the chown
on the conda folder (here) takes about 8 min to complete. Probably this has not been an issue before, but only occurs now after the meltdown patches? I guess this might also lead to considerable slowdowns in CI. I wonder if there is a way to avoid the chown?
There may be good reason for this, I noticed that the linux-anvil-aarch64
and linux-anvil-ppc64le
Dockerfiles don't install make
but the linux-anvil-comp7
Does.
Is that by design?
https://hub.docker.com/r/condaforge/linux-anvil/builds/bo2anargh9vx5zchqrdwmrr/
Removing intermediate container 67f92b03cba2
Step 4 : RUN yum update -y && yum install -y bzip2 gcc-c++ git make patch tar which libXext-devel libXrender-devel libSM-devel libX11-devel mesa-libGL-devel && yum clean all
---> Running in 467dbb0d7283
Loaded plugins: fastestmirror
Setting up Update Process
�[91mError: Cannot retrieve repository metadata (repomd.xml) for repository: extras. Please verify its path and try again
�[0mCould not retrieve mirrorlist http://mirrorlist.centos.org/?release=6&arch=x86_64&repo=extras&infra=stock error was
14: PYCURL ERROR 22 - "The requested URL returned error: 500 Internal Server Error"
Removing intermediate container 467dbb0d7283
The command '/bin/sh -c yum update -y && yum install -y bzip2 gcc-c++ git make patch tar which libXext-devel libXrender-devel libSM-devel libX11-devel mesa-libGL-devel && yum clean all' returned a non-zero code: 1
Issue: It appears that there's no system wide Perl
installation in condaforge/linux-anvil-comp7. This is not really a big deal since Perl
is installed in the base environment:
[conda@7cf862609754 ~]$ which perl
/opt/conda/bin/perl
However, I found out that some applications look for perl in specific locations, namely /usr/bin/perl
. For e.g: The cuda-toolkit installer expects to find Perl
in /usr/bin/perl
: conda-forge/cudatoolkit-dev-feedstock#11
/opt/conda/pkgs/cudatoolkit-dev/cuda_10.0.130_410.48_linux: ./cuda-installer.pl: /usr/bin/perl: bad interpreter: No such file or directory
I am wondering whether there exists a simple fix here or whether this would require having a system wide Perl installation as part of the docker image?
Since a recent update our conda package building pipe line broke.
Specifically, it fails on:
➜ ~ docker run -it condaforge/linux-anvil
(root) [conda@9e93de846f43 ~]$ yum install -y devtoolset-2-gcc-gfortran
Loaded plugins: fastestmirror, ovl
ovl: Error while doing RPMdb copy-up:
[Errno 13] Permission denied: '/var/lib/rpm/Conflictname'
You need to be root to perform this command.
+ /usr/bin/sudo -n yum install -y mesa-libGL mesa-dri-drivers libselinux libXdamage libXxf86vm libXext libXfixes libXinerama libXrandr libXcursor libXcomposite libX11 libXi
Loaded plugins: fastestmirror, ovl
Setting up Install Process
Determining fastest mirrors
Error: Cannot find a valid baseurl for repo: base
YumRepo Error: All mirror URLs are not using ftp, http[s] or file.
Eg. Invalid release/repo/arch combination/
removing mirrorlist with no valid mirrors: /var/cache/yum/x86_64/6/base/mirrorlist.txt
See build pipelines in conda-forge/hugin-feedstock#7
In a recent test I found that the glibc version in CF's CUDA docker images is too old (2.12), and it is unable to support new CUDA libraries such as cuTENSOR which needs 2.14+ (conda-forge/cupy-feedstock#57).
Quoting @jjhelmus from conda-forge/cupy-feedstock#57 (comment):
The CUDA Toolkit 10.0 release notes indicated that support for RHEL 6.x was deprecated and may be dropped in a future release. I would not be surprised if cuTENSOR did not support RHEL 6.x.
RHEL 6 / CentOS 6 use glibc 2.12, the later is the base for the
conda-forge/linux-anvil-cuda:10.0 image
.
Several of my builds have started failing with the most recent condaforge/linux-anvil
docker image (for example, see here).
Traceback (most recent call last):
File "/opt/conda/bin/conda-build", line 11, in sys.exit(main()) File "/opt/conda/lib/python3.6/site-packages/conda_build/cli/main_build.py", line 413, in main execute(sys.argv[1:]) File "/opt/conda/lib/python3.6/site-packages/conda_build/cli/main_build.py", line 404, in execute verify=args.verify) File "/opt/conda/lib/python3.6/site-packages/conda_build/api.py", line 188, in build need_source_download=need_source_download, config=config, variants=variants) File "/opt/conda/lib/python3.6/site-packages/conda_build/build.py", line 1913, in build_tree notest=notest, File "/opt/conda/lib/python3.6/site-packages/conda_build/build.py", line 1053, in build is_conda=m.name() == 'conda') File "/opt/conda/lib/python3.6/site-packages/conda_build/environ.py", line 812, in create_env execute_actions(actions, index, verbose=config.debug) File "/opt/conda/lib/python3.6/site-packages/conda/common/io.py", line 441, in decorated return f(*args, **kwds) File "/opt/conda/lib/python3.6/site-packages/conda/plan.py", line 392, in execute_actions execute_instructions(plan, index, verbose) File "/opt/conda/lib/python3.6/site-packages/conda/plan.py", line 599, in execute_instructions cmd(state, arg) File "/opt/conda/lib/python3.6/site-packages/conda/instructions.py", line 64, in PROGRESSIVEFETCHEXTRACT_CMD progressive_fetch_extract.execute() File "/opt/conda/lib/python3.6/site-packages/conda/core/package_cache.py", line 590, in execute raise CondaMultiError(exceptions) conda.CondaMultiError: [Errno 21] Is a directory: '/opt/conda/pkgs/setuptools-38.4.0-py36_0'
I am using the most recent version of conda (as obtained using conda update -n root conda conda-build
), and I think the packages inside the /opt/conda/pkgs
folder are somehow too old for conda to handle them gracefully. Running rm -rf /opt/conda/pkgs/*
before conda build .
seems to fix the issue.
Maybe rm -rf /opt/conda/pkgs/*
should be included in the Dockerfile
? This would have an additional advantage of making the docker image smaller.
Currently the images just have a "latest" tag, and over on bioconda we recently had builds failing due to changes in the linux-anvil
image we weren't aware of. Pinning to a tagged image would help this sort of thing in the future. @jakirkham suggested git sha as tags, which for our purposes would be great
We need a CentOS 7 image for building CUDA, especially CUDA 11.0. I am sure it's on the roadmap, just opening an issue here for cross referencing from other places 😁
So centos bundles binutils, and the whole OS depends on them.
If you want to make sure you don't use them, I would recommend adding a one liner that just removes them.
rm -f /usr/bin/{ar,addr2line,as,c++filt,dwp,elfedit,gprof,ld,ld.bfd,ld.gold,nm,objcopy,objdump,ranlib,readlef,size,strings,strip}
This is going to make many builds frustrating when they fail on a missing ar
when the build is almost over....
Is make necessary in the docker image?
It seems we can always add it to the conda dependencies right?
Right now it doesn't seem there is a GitHub team for this repo. Seems like it would be easier for org maintenance and communication if there was one. Guessing docker-images
would be the team name. Though we may need to update it if we rename the repo ( #25 ).
As setuptools
has now made it into the conda-build
requirements ( conda-forge/conda-build-feedstock#101 ), it would be nice to drop it from this install line. However we will likely need to wait for a few conda-build releases until things have had a chance to settled down a bit more.
Appears they are running into an issue getting Conda 4.8.5. Not sure why that version is being used
Collecting package metadata (current_repodata.json): ...working... done
1386
Solving environment: ...working... failed with initial frozen solve. Retrying with flexible solve.
1387
Solving environment: ...working... failed with repodata from current_repodata.json, will retry with next repodata source.
1388
1389
ResolvePackageNotFound:
1390
- conda==4.8.5
https://github.com/conda-forge/docker-images/runs/3648769402
Seems this repo is using GitHub issue/PR templates designed for feedstocks, which doesn't make much sense here. Would be nice to disable them here short term and possibly update them later.
It appears that #181 did not fix the problem, see conda-forge/cutensor-feedstock#12 (comment).
As our current docker image is being moved to community maintenance ( #5 ), we should probably update the MAINTAINER
tag to reflect this. Unfortunately, I don't think the MAINTAINER
tag responds well to multiple authors and maybe that is not the way for us to go anyways. I propose the following adjustment. Feedback welcome.
MAINTAINER conda-forge <[email protected]>
This build is pretty hefty so we should move it to azure.
Since we include some components we may want to cache inside the images -- it might be a good idea to build the docker containers daily instead of ad-hoc. This should be able to reduce the time that linux builds take to build by avoiding needing to solve and download a subset of the packages.
The license file added here was MIT. Though typically we use BSD 3-Clause with source code. Are we happy with MIT license here or do we wish to switch it to BSD 3-Clause?
cc @conda-forge/core
It seems many of our other maintenance repos start with a conda-forge-
prefix. Wondering whether this one should follow that trend.
Am interested in seeing how we can standardize the images here a bit and consolidate maintenance effort. In particular am curious how we can make a multi-arch image. There's a good blogpost on this topic. It would be good to determine what the current issues prompting the different images we have are and how we address those issues in a general way that benefits all of the images here.
Would it make sense to not only include mamba (And where necessary mambaforge) in the image but also build it with that?
Happy to create the PR but wanted to get some opinions first.
The layer created here has ~1.4GB size:
docker-images/linux-anvil-comp7/Dockerfile
Lines 31 to 48 in 6152d61
The image for circle-ci (centos I believe) has no unzip command available. I think it would be useful to add it. E.g. github downloads are zip by default (and it is not that easy to find how to get a tarball).
There are currently at least two proposals for build images:
conda-forge/staged-recipes#97
and
conda-forge/staged-recipes#97 (comment)
Both are based on CentOS 5.11. The former is a stripped-down version of the manylinux docker image that includes Miniconda. The latter builds GCC 5.2 instead of packaging the RedHat devtools (GCC 4.8) that manylinux (and many others) depend on.
This issue's discussion should cover:
now that we also have mambaforge
, it would be great to add a docker image to the docker hub.
The Dockerfile is the same of miniforge
, replacing the name with mamabaforge
.
I get the following error when I try to run yum update
using the latest docker image:
$ docker run --rm condaforge/linux-anvil:latest yum update
bash: cannot set terminal process group (-1): Inappropriate ioctl for device
bash: no job control in this shell
Loaded plugins: fastestmirror, ovl
ovl: Error while doing RPMdb copy-up:
[Errno 13] Permission denied: '/var/lib/rpm/Sha1header'
You need to be root to perform this command.
The DockerHub builds are having some sort of checksum error when installing some things. As this is not happening with CircleCI and I am unable to reproduce it locally, it seems like a DockerHub issue. Added the following upstream issue ( docker/hub-feedback#675 ). Let's hold on new changes for a bit.
If DockerHub remains unstable in these sorts of ways (sadly not the first issue I have encountered with automated builds on DockerHub), we could try quay.io or we could just push images from CircleCI. If we try the former, we can mirror images on DockerHub to avoid having to switch a bunch of links.
We have a Docker image on Docker Hub called conda-forge-x86_64
. It hasn't seen any pushes in 5mos, it has only a handful of pulls (17), was manually pushed, and AFAICT is unused. Few questions.
Please consider renaming the jnlp-slave
Docker config and the corresponding image name(s). Some background: https://bugs.python.org/issue34605. Thank you.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.