federatedcloud / nixtemplates Goto Github PK

There are numerous performance improvements that this could entail:

1. No need to initialize the nix repository upon insantiation of the container for the first times
2. Fewer checks each time the container is started (at least, I think so).
3. Packages could be shared between users on the VM, but this requires nix be installed with setuid accordingly - probably the standard nix installer handles this, but am unsure - certainly it is handled appropriately in NixOS
4. No need to use a beefy overlay image (may not need overlay at all).

Of course, there are some places where this won't work:

1. Where we don't control the nodes - e.g., Stampede or other large HPC systems. We can't mess with setuid there.
2. Where we really want to provide image snapshots for ultimate reproducibility (still, this comes in pairs: the base image and the overlay image).

https://github.com/federatedcloud/NixTemplates/blob/develop/Dockerfile-OpenMPI#L1

To maintain similar functionality, need to switch to some sort of text templating system that can generate our Dockerfiles for us.

Could use Data.TemplateString for a typed and relatively light alternative relying on NodeJS, along with something like Snail for scripting, but see these comments

Not exactly an issue for this repo, but here it is. Closest I've gotten is to try to use the host singularity, owned by root:

 singularity run --contain -B /tmp/shubtmp:/tmp -B /usr/local:/host_local --overlay nix-overlay.img shub://federatedcloud/NixTemplates:nix_alpine_openmpi_6796a60398bb890002e7010593c14cf3731613a1

Then I try to do:

/host_local/bin/singularity run shub://singularityhub/hello-world

But need to use a system that has /bin/bash installed ... or a singularity image where the user can do sudo. To be continued.

This could be useful for allowing MPI installed int he container to be run "on the host" by making the mpirun user's default shell the container housing MPI (which would need to be configured by the VM-provisioning software of choice).

Already built nix_alpine_base per README, cannot build nix_alpine_openmpi

:# source Docker/OpenMPI/build.sh
NIX_OMPI_IMAGE is nix_alpine_openmpi:53bf75e88ccbd0b9719d2c63ffea944e344e1bcb
Sending build context to Docker daemon 146.9kB
Step 1/35 : ARG BASEOS
--->
Step 2/35 : FROM nix_${BASEOS}_base:11a12ceb5c9cdedd3816acc783d193eeb1fe7fa6
pull access denied for nix_alpine_base, repository does not exist or may require 'docker login'
Unable to find image 'nix_alpine_openmpi:53bf75e88ccbd0b9719d2c63ffea944e344e1bcb' locally
Error response from daemon: pull access denied for nix_alpine_openmpi, repository does not exist or may require 'docker login'
Error response from daemon: No such container: nix_alpine_openmpi_53bf75e88ccbd0b9719d2c63ffea944e344e1bcb_TEST
Error response from daemon: No such container: nix_alpine_openmpi_53bf75e88ccbd0b9719d2c63ffea944e344e1bcb_TEST
Error response from daemon: No such container: nix_alpine_openmpi_53bf75e88ccbd0b9719d2c63ffea944e344e1bcb_TEST

:# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
nix_alpine_base 53bf75e 5f19395e57df 9 minutes ago 279MB
alpine 3.7 3fd9065eaf02 5 months ago 4.15MB
jupyter/all-spark-notebook latest bbfb8aad625b 8 months ago 5.24GB

cat /etc/issue
Ubuntu 16.04.4 LTS \n \l
docker --version
Docker version 17.09.0-ce, build afdb6d4
git log | head -n 1
commit 53bf75e

As done here

I do not recall where this came from or what it does.

This will be somewhat meta, as ideally we should be able to just build these images on any system with docker and/or singularity, but we're starting to see some dependencies in the build process:

Here is a list (update as necessary):

• spython (from singularity-cli) - used occasionally to generate Singularity recipes from Dockerfiles. Probably not necessary to include, but might as well if we build this.
• GNU gettext (for envsubst) - used while Singularity doesn't support environment variable expansion.

This would need to be a non-interactive script used by non-base images (e.g. the OpenMPI image).

Even though we are doing dev work in 'develop', if we merger in commits to 'master', a singularity image will be built for each commit in 'master', with some intelligent caveats:

If a user has a repeat tag, then the file with more recent changes (timestamp) is built. Only Singularity files that have been changed since the last commit will be built with Automatic Building in this fashion. If you want to go back int time to a particular commit, you should use a Manual Build. This means that if your push to Github has 10 changed Singularity recipe files, all 10 will be added to the queue to build, and your collection is allowed one active build at a time.

This is probably reasonable, but we should discuss.

Currently .dockerignore ignores *.img and *.simg, however, most of these still seem to get picked up, as indicated by the size of the build context and the tool docker-show-context
. Not sure what is going wrong.

Specifically all Singularity recipes (and by extension / call for simplicity, Dockerfiles) should be moved to the repo root. See singularityhub/singularityhub.github.io#140 for context.

The build part can be accomplished by copying the repo dir to /tmp/<randomId>, checkout out appropriate commit, then building the base image.

As discussed at https://stackoverflow.com/a/50747830/3096687 and pure-eval mode: nix-shell --pure --pure-eval ... with the nix expression embedded in a simple shell script should work.

nix-shell -p <pkg_attribute_name> -I /path/to/nixpkgs

and nix-env?

Notes from output:

/OpenMPI$ ~/spython recipe Dockerfile > SingTemplate
WARNING ARG is not supported for Singularity! To get BASEOS
WARNING in the container, on host export SINGULARITY_BASEOS
WARNING ARG is not supported for Singularity! To get BASEDIR
WARNING in the container, on host export SINGULARITY_BASEDIR
WARNING ARG is not supported for Singularity! To get ADDUSER
WARNING in the container, on host export SINGULARITY_ADDUSER
WARNING $BASEDIR/config.nix doesn't exist, ensure exists for build
WARNING $HOME/.config/nixpkgs/ doesn't exist, ensure exists for build
WARNING $BASEDIR/dev-env.nix doesn't exist, ensure exists for build
WARNING $ENVSDIR/ doesn't exist, ensure exists for build
WARNING Utils/persist-env.sh doesn't exist, ensure exists for build
WARNING $ENVSDIR/ doesn't exist, ensure exists for build
WARNING $BASEDIR/ssh/config doesn't exist, ensure exists for build
WARNING ${SSHDIR}/config doesn't exist, ensure exists for build
WARNING $BASEDIR/ssh/id_rsa.mpi doesn't exist, ensure exists for build
WARNING ${SSHDIR}/id_rsa doesn't exist, ensure exists for build
WARNING $BASEDIR/ssh/id_rsa.mpi.pub doesn't exist, ensure exists for build
WARNING ${SSHDIR}/id_rsa.pub doesn't exist, ensure exists for build
WARNING $BASEDIR/ssh/id_rsa.mpi.pub doesn't exist, ensure exists for build
WARNING ${SSHDIR}/authorized_keys doesn't exist, ensure exists for build
WARNING $BASEDIR/default-mca-params.conf doesn't exist, ensure exists for build
WARNING ${HOME}/.openmpi/mca-params.conf doesn't exist, ensure exists for build
WARNING $BASEDIR/mpi4py_benchmarks doesn't exist, ensure exists for build
WARNING ${HOME}/mpi4py_benchmarks doesn't exist, ensure exists for build
WARNING $BASEDIR/entrypoint* doesn't exist, ensure exists for build
WARNING $ENVSDIR/ doesn't exist, ensure exists for build
WARNING $BASEDIR/default.nix doesn't exist, ensure exists for build
WARNING $ENVSDIR/ doesn't exist, ensure exists for build

As done here

These files largely overlap, but need some extra logic depending on which build is being done.

Should be executable:

ls -last /.singularity.d/runscript*   
     0 -rwxr-xr-x    1 root     root           489 Jul 30 19:47 /.singularity.d/runscript  
     1 -rw-rw-r--    1 root     root           732 Jul 30 19:37 /.singularity.d/runscript-nixbase 

Since the version of MPI in the container should be <= the version on the host to be able to communicate (ref), we should pin versions, by default, and a sensible version supported by most major HPC centers should be chosen. I assume this will be 3.0 or even 3.1; for our own systems it should be easy to do.

In fact, for our own systems, we should first investigate just using mpirun from singularity to launch a singularity container, so we can avoid having to do this pairing up at all on our systems.