Entrypoint is a great feature... However, for the untrained docker user (like me) it can be extremely frustrating to not know how to "get into" the container and check things out, for example because the entrypoint command fails and then the container stops and we don't know why.
Because of that I wrote a blog about this and how to by-pass the entrypoint: https://bcrf.biochem.wisc.edu/2019/10/31/docker-notes-bypass-entrypoint/

which contains 2 references to other blogs:

• How to properly override the ENTRYPOINT using docker run:
  https://oprea.rocks/blog/how-to-properly-override-the-entrypoint-using-docker-run/
  (archived 11NOV 2019: bit.ly/2QFlGne)

• Docker RUN vs CMD vs ENTRYPOINT:
  https://goinbigdata.com/docker-run-vs-cmd-vs-entrypoint/
  (archived 25MAR 2019: bit.ly/35pg8Bj )

This may be the only way to get into a container and use a program even if the entrypoint command fails.

Originally posted by @jsgro in #41 (comment)

The title of this lesson is fairly generic and it is, I think, equally applicable to the new singularity-introduction lesson that has just been added to the carpentries-incubator.

Indeed, the material in the singularity-introduction lesson was originally developed and taught as an extension to this material in this workshop.

While the Singularity lesson has been spun out into a separate set of material, to maintain a link with this material (and support the case where they may be taught together as part of a workshop) it has been given the title "Reproducible computational environments using containers: Introduction to Singularity".

Is it worth considering a renaming of this lesson to "Reproducible computational environments using containers: Introduction to Docker", or, to avoid confusion should the Singularity lesson be renamed to something different, such as "Introduction to Singularity"?

See issue 3 in the singularity-introduction repo for further information/discussion.

Met up with @aturner-epcc and @jcohen02 today and we discussed that the lesson needs more exercises. Adding this as a placeholder to look for places where we might add more exercises.

I'd like to update the citation link but need some advice. Curriculum Dev Team (@fmichonneau, @ErinBecker, @zkamvar), what is the best way to do this? Should I do a major release and publish with zenodo?
Thanks!

Windows troubleshooting advice?

Which parts of the lesson to skip / emphasize.

Hi @dme26!

I added a section to the lessons where I showed them how to interactively work in a docker container. While it int as reproducible of a way to build images, it is often still useful. I often work interactively with a container while I'm writing my dockerfile to check that the commands will work as expected. Someone in the workshop suggested maybe adding a section that mimics that workflow.

It is at the bottom of this lesson, the section called "Creating an image interactively". It might need to go elsewhere in the lessons. I also think it might need to be a better example. https://uw-madison-datascience.github.io/docker-introduction/05-creating-container-images/index.html

Let me know what you think.

I (Christina) am guilty of using image vs container interchangeably. Should check the whole lesson for correctness. :P

I made this sequence of slides to illustrate the different ways to "run" a container.

https://docs.google.com/presentation/d/1nqF53xAeqgbONvTkuSs7Uqf5PicVZ0U1XYLnIu2Y-0k/edit?usp=sharing

Is this useful to add to the lesson, either as slides in the files folder or a single diagram in the lesson (or both)?

I would change "Creating Containers" to "Using Containers" -- we talk about actually creating new container images with a Dockerfile in a later episode.

The lesson talks about tags in the image names OWNER/REPO:TAG but the docker tag syntax has an entirely different meaning -- it's like the whole label or name for the container, not just the version tag. We should probably rename the :TAG to something else to keep those from being confused. :(

Hi there,

I was just working through the "Creating your own container images" episode, but I ran into an issue with the following command: /# apk add --update python py-pip python-dev.

I got the following error message:

ERROR: unsatisfiable constraints:
  python (missing):
    required by: world[python]
  python-dev (missing):
    required by: world[python-dev]

Everything else with the alpine image has worked so far! Additionally, the next exercise, installing R, worked just fine.

Hi,
I just wanted to point out something really useful when running e.g. bioinformatics tools with docker containers.

This could fit here Creating advanced containers But on the other side, it will eventually make the duration of the episode a lot longer.

It relates to the fact that if you run tools with docker containers which create output files outside the docker, so using the -v flag on Linux, you might create files with the owner root (it depends how the recipe is created though).

When using -u="$(id -u):$(id -g)" as additional flag running the container, you can avoid this surprise.

Maybe this is too advanced for your current course but I find it great that there is this run-time flag.

@ErinBecker @fmichonneau What is the best way to add maintainers to this repo? We would like to add @ChristinaLK and maybe others in the near future.

In episode
Containers used in generating this lesson
this sentence appears.

In the preceding output, you see Docker downloading the image for Jekyll, which is a tool for building websites from specification files such as those used for this lesson. The line jekyll serve indicates a command that runs within the Docker container instance. The output below that is from the Jekyll tool itself, highlighting that the website has been built, and indicating that there is a server running.

This seems to refer to the jekyll serve command rather than the make serve command.

In Exploring and Running Containers in the challenge Practice Makes Perfect, it wasn't obvious to me that /proc/version was the file to look at.

For someone new to linux, they might immediately start looking for the /etc/os-release file. In my case, I just gave up and looked at the answer, which arguably made it less useful for me than if there had been a hint about the file name.

One of my learners from a couple of weeks ago reported back that he made an image for his work recently! 😄

He mentioned that he didn't really know much about layers until he was constructing this image. While I off-handedly mentioned layers during the workshop, he suggested maybe we add a 15 min section about layers. I also think it would be a good idea. I'm wondering if we could add a conceptual explanation and also build up a dockerfile with one layer at a time, showing that the previous layer exists and that it doesn't get re-built (unless you change something above)?

My identifier dme26 should probably be removed from code listings. I don't actually mind if it stays there, but the "replace with your ID" pattern must be common across, e.g., Carpentry lessons, and this lesson should ideally adopt good/better/best practice in this regard.

In episode "Creating More Complex Container Images", there is this line:

Take a look at the Dockerfile. It looks similar to the one we used before, but it has an additional line with the COPY keyword.

This implies I should see similar content in the Dockerfile to the file created in the previous exercise. But I see:

FROM <EXISTING IMAGE>
RUN <INSTALL CMDS FROM SHELL>
RUN <INSTALL CMDS FROM SHELL>
CMD <CMD TO RUN BY DEFUALT>

When I ran this with my test group, they thought we should show the python docker hub page instead of R since we use python for the rest of the lesson. I did when I taught it but I think it might be helpful to add more examples of dockerhub pages to the lessons to give them more than one example so maybe we could show both?

Need to add @ChristinaLK @aturner-epcc and @jcohen02 to maintainers in the Readme

Hi all! Please let us know if you are teaching this lesson by copying the info below and replying to this issue.

Name:
Institution:
Date Taught:
Was this taught as a stand alone workshop or as part of another workshop / conference?
What feedback do you have on the lesson?
Anything else you'd like to share?

Please have your learners fill out Docker post-workshop survey after the workshop.

Want to add:

• how to find images
• naming conventions + tags
• an exercise asking people to find an image to explore + to do so

Hopefully will have this done by next week.

08-reproducibility -- solution to the "positives and negatives" exercise doesn't have titles above each list the way it should.

05b-advanced-containers -- "no such file or directory" exercise/callout is missing the right blockquote tag

05b-advanced-containers -- "copying alternatives" callout is missing the right blockquote tag

Thanks @ErinBecker for setting up this repo!

@dme26 I think first thing we need to:

• Migrate the episodes to this new repo
• Migrate the issues to this new repo
• Update the readme
• Homepage introduction

from https://github.com/dme26/docker-introduction

Let me know if there other other first things we need to do

The current contents of "Containers used in generating this lesson" may not be ideal for every audience.

persona: Suppose a scientist has some small projects that they want to create a portfolio of. In that case it would be nice to be able to convert a README.md (or similar) to a website using pandoc.

I propose a chapter where we create a new project from scratch on Github. The project will only contain a readme file. Then we create a Github action that uses the pandoc/core docker container to convert the markdown to HTML and uploads the result to the gh-pages branch. This can be done in a minimalist fashion where a participant can understand each step in the process.

It should be noted that pandoc instances in package managers (why not just apt install it?) are often outdated.

questions

• How do I use docker from a Github Action?

objectives

• Awareness of possible use of Docker on cloud based source control platforms.
• Understand how Docker is called from a Github Action

keypoints

• Setting up a Github Action with Docker can automate every task.
• Docker gives ultimate flexibility in customizing and optimizing Github Actions (and similar concepts on competing platforms).

content

1. Create project with README
2. git clone and run pandoc/core image locally to show its function
3. Create Github Action
4. Watch magic happen

An example of the resulting project: https://github.com/jhidding/readme-pages

remarks

Pros: The advantage over the existing lesson is that the participants won't need to download a prepared project. This helps in empowering them to start using these methods in their own work. Using Github for this lesson ties in nicely with the Carpentries lessons on source control.
Cons: Using Github for this lesson widens the vendor lock-in on a monopolistic institution ;)

PR will be coming in the near future

The first episode talks about some of the rationale for using containers - would it be helpful to introduce specific use cases? Like:

• run an existing container on your computer, to either use the tool inside or to test it out
• build your own container to share with your group or run on a shared system

This is kind of touched on when we talk about reproducibility and portability, but we never connect the lesson material with these kinds of specific examples.

The example used in the "Containers used in generating this lesson" episode (_episodes/07-containers-for-continuous-integration.md) currently seems to be broken.

When you try to run the example using the command provided:

docker run --rm -it -v ${PWD}:/srv/jekyll -p 127.0.0.1:4000:4000 jekyll/jekyll:3.7.3 make serve

You get an error:

Could not locate Gemfile or .bundle/ directory
make: *** [Makefile:26: serve] Error 10

It seems that the Makefile is trying to run bundle exec jekyll serve which relies on the various gem dependencies being installed by bundler either in a global or local environment. For example, by convention, these dependencies would often be installed in the vendor/bundle subdirectory under the project.

Since we're using the jekyll docker container, we should have all the required dependencies installed globally and bundler shouldn't be required.

I was sure this was working originally and it looks like the jekyll command in the Makefile was changed in commit 4e1e6b0 from jekyll to bundle exec jekyll.

The options here are:

• Add an instruction to the lesson requiring the installation of the dependencies before running the docker command (e.g. run bundle install --path vendor/bundle)
  OR
• Change the command in the Makefile back to jekyll, although I'm not sure what unintended consequences this may have

At the same time, I'd recommend increasing the jekyll version to something more recent. For example, we could use 3.9.0 as listed in the GitHub Pages dependency versions as the version currently used.

I think showing how to use --help for a lot of the commands would be a good idea, especially at the beginning when introducing docker image

We should add something to the lesson about the new Docker Hub limits, especially if people are using Docker containers for reproducibility purposes.

Hi all! @aturner-epcc , @ChristinaLK, @jcohen02, @dme26

I think we should meet to discuss goals for the lesson. With some polishing, we could get this out of beta and it would be helpful to talk about it as a group.

I know we are a difficult bunch to schedule and the holidays are coming up so I'd like to use this issue as a placeholder to get us discussing when we can meet.

I propose we plan to meet in Feb or March. Reply below with...

• Are there any weeks that are not possible for you during that time?
• What UTC time range could we possibly meet in for you.

My answers:

• No particular blocked dates yet but March might be better than Feb as I'm hosting a research Bazaar.
• My UTC time range is 13 UTC - 3 UTC.

In Lesson 9 -
https://carpentries-incubator.github.io/docker-introduction/07-containers-for-continuous-integration/index.html

I tested the command below in both macOS and Windows10:
docker run --rm -it -v ${PWD}:/srv/jekyll -p 127.0.0.1:4000:4000 jekyll/jekyll:3.7.3 make serve

In both cases there is a series of errors.
IN THE END the correct way to do this is changing the 2 things: the -p option and the requested command. The following should work for all:

docker run --rm -it \
  --volume="$PWD:/srv/jekyll" \
  -p 4000:4000 jekyll/jekyll:3.8.5 \
  jekyll serve

Inspired by https://martinpeck.com/jekyll/blog/containers/2019/02/02/build-and-deploy-jekyll-with-docker-part1/

The tricky part is that the directory contains a Makefile that we'll run within the container... but itself will make a container, hence the HTTP never gets to the Mac or Windows PC... (as far as I understand)

Changing make serve to jekyll serve makes the difference: now the HTTP server will be made within the docker container just started.

NOTE: there was another line --volume="$PWD/vendor/bundle:/usr/local/bundle" \ which I removed as I was getting time-stamp error (a Gemfile.lock was created as well as a vendor directory that made it difficult for me to test various commands...)

@fmichonneau has updated the lesson template to enable support of Incubator lessons!

This update:

• Creates a top-of-page banner to indicate that this lesson is in the incubator (and not an official Carpentries lesson). This banner is in addition to the existing "lesson life cycle" banner that appears on some official Carpentries lessons (for example the geospatial lessons) to indicate whether lessons are pre-alpha, alpha, beta, or stable.

• Changes the license information to make it clear the copyright is not held by The Carpentries or any of its lesson programs.

• Replaces The Carpentries (or SWC/DC/LC) logo in the upper left hand corner of the template with The Carpentries Incubator logo.

@fmichonneau and I are happy to do the heavy lifting of bringing this template into your lesson repository (or helping you with the logistics if you'd like to try it yourself). Once this has been done, your lesson will appear on a "Community Contributed Lessons" page on The Carpentries website!

Please respond to this issue to let us know if you approve these changes. Please also indicate whether you would prefer to pull in the template update yourself, or would prefer @fmichonneau and I to take the lead on that. Looking forward to posting your lesson publically and getting the word out about this awesome resource!

There might be a situation where a container is already running, but we cannot turn it off at the moment for various reasons. (For me it was because I had tested/added lengthy modifications but I had --rm in the original command so exiting would have destroyed my changes.)
I also wrote a blog on this: https://bcrf.biochem.wisc.edu/2019/11/01/docker-notes-shell-access-to-running-container/

There are 2 concepts to think about:

1. entering a container that is already running
2. the container name

The container name is randomly given (and can be seen with "docker ps") but can also be specified by the user using the "--name myname" method so that the name is always the same.

The name can then be more easily used to run a command from a separate terminal to enter a shell of that container which here could be: docker exec -it myname /bin/bash

I suppose that this would be useful information.

There are multiple layers of "difficulties" that are not yet mentioned in the lesson but might be too much for this course at this time. These ideas come from my self-teaching Docker last year to implement my tutorial https://bcrf.biochem.wisc.edu/docker-beginner-for-biologists/

Namely: difficulties raised by docker that would run graphical software:

• How to use an X11 program
• How to map and use a Web serving address (useful for RStudio)

There are examples and exercises on the 4th lesson here:
https://static-bcrf.biochem.wisc.edu/tutorials/docker/Docker_04.html

I only tested those on Mac, but with Xwin32 or MobaXterm it might be possible to run X11 on Windows as well... But this is certainly complicated.

Originally posted by @jsgro in #56 (comment)

Hello Docker enthusiasts,

I am running a VIP workshop on Docker for our Amsterdam science park study group core members. The idea is to self-teach us Docker during an afternoon where we all sit down and go through your nice lesson material. I'll probably be able to spot a few issues and improvements.

Two questions:

• Do you agree to add an episode on Singularity and make a singularity branch & a pull request?
• Do you have any wish list or already existing materials to guide this new episode section?

Thanks
Marc

It probably makes sense to augment lessons with information regarding what happens for each of the main mechanisms for installing Docker, as they may well have different behaviour.

I was worried it might be a bit confusing when we name the directory and image something with container in the name so I changed the names of the images I built to try to avoid this.

Sorry to bother you with what might feel a trivial thing... but it could be frustrating in some cases.

On the lesson pages there are currently 2 (two) ways that the "current directory" is referenced: $PWD and also ${PWD}. They both work on macOS, but $PWD will Fail on Windows PowerShell (I just tried again.)

In fact when I learned about this I learned it with parenthesis as $(pwd) which works on macOS and Linux but not on Windows. (I just verified on Windows.)

Also, even though I am not able to test this, I would think that UPPercase formats would fail in Linux because of case-sensitive shell, while the Mac is (usually set to be) case insensitive.

Therefore the most common format that should work on macOS, Linux and Windows PowerShell is likely to be: ${pwd}` with curly brackets and lower case.

I did note the cmd.exe format "%CD%" which completely different but is a useful addition for a few people.

From my understanding we aren't using the container_playground folder anymore. Please let me know if you are still using that folder somewhere. Otherwise, I think we should remove it from the prereqs box.

I suppose that this idea might not fit with the carpentries/git-hub-pages template without modifications, but my way of showing this was to use a different background color for when we are running the docker container from "outside" rather than "dive into it" so that we are actually looking inside the container file system. You can see how this looks in my Docker lessons from last year:

DOCKER – BEGINNER FOR BIOLOGISTS
https://bcrf.biochem.wisc.edu/docker-beginner-for-biologists/

You can take a quick look at this slide for the idea with 3 colors:
https://static-bcrf.biochem.wisc.edu/tutorials/docker/docker_ioslides_01.html#10

This slide with 4 colors (one for alternate Terminal)
https://static-bcrf.biochem.wisc.edu/tutorials/docker/docker_ioslides_04.html#22

Originally posted by @jsgro in #53 (comment)

Hello! This issue was raised at the recent NZ Carpentries Connect unconference See Line 159 - I’m tagging the following partners in crime who agreed we should submit this idea to the lesson: @dme26, @CallumWalley. Also, thanks to @mpfl for drafting the background on this, because I am not an HPC specialist, but an enthusiastic enabler!

Some infrastructure providers (e.g. HPC providers) do not run Docker due to security concerns in multi-tenanted environments, and prefer to run alternative container solutions like Singularity. The biggest problem is that Docker runs containers with root privileges by default.

Furthermore, some Linux flavours have moved away from using the Docker daemon by default and are using other container daemons like containerd.

Could there be a way to model a Customise This! component throughout the lesson to better support instructors customising this lesson for their own local environments? For example: including a checklist in the readme for instructors to be able to specify which container daemon to use, as well as local security requirements for the teaching environment.

@chendaniely brought up the question of when you would use these two different options in a Dockerfile

This link has some clarification I found helpful.

Maybe we could add a section or call out in the "Creating More Complex Container Images" lesson to address this?

In episode
Creating More Complex Container Images
and callout
Overriding the ENTRYPOINT

The interactive call changes from -it to -ti. To build muscle memory and help recall, maybe it could be the same order in both cases.

Lesson 3

Exploring and Running Containers

https://carpentries-incubator.github.io/docker-introduction/03-running-containers/index.html

Run the Alpine Docker container
Try downloading and running the alpine Docker container. You can do it in two steps, or one. What are they?

What happened when you ran the Alpine Docker container?

$ docker run alpine
Probably nothing! That’s because this particular container is designed for you to provide commands yourself. Try running this instead:

The result is:

$ docker run alpine
Unable to find image 'alpine:latest' locally
latest: Pulling from library/alpine
df20fa9351a1: Already exists
Digest: sha256:185518070891758909c9f839cf4ca393ee977ac378609f700f60a771a2dfe321
Status: Downloaded newer image for alpine:latest

Issue brought up at the March 2021 maintainer meeting. We decided that @dme26 and @jcohen02 would move it to extras and then work on a redesign. This issue can be closed when it is moved to extras.

I had 3 extra laptops just in case there were errors I couldn't solve with docker on the fly. It might be my lack of experience with windows but all 3 of them ended up going to people with windows and a majority of the errors were on windows machines.

I was thinking maybe we could try to have them to use the quick start terminal or a specific shell to try to standardize the approach so we can rule those out as possible reasons why some folks are getting errors?

@tobyhodges we talked a bit about this when we met last week but it came up again in my meeting today with @aturner-epcc and @jcohen02. We thought would be nice to have a survey for learners and maybe one for instructors to get feedback from the individuals using the lesson.

@tobyhodges Were you thinking something that could be use for any incubator lesson? If so, anything we can do help construct that survey?

I've not used docker desktop for windows and haven't tried it for the whole workshop yet. Think we should try it out and then update the install directions.

I've just read through the training (nice!) and have some comments and could potentially contribute. I'll start with a general description of a specific issue.

The use of the terms image and container could be more precise and consistent. I think the accepted/defined usage is that the image is the static artefact and the container is a running instance (which started as/from a particular image). This may link with Section on Layers

Also, the description of 'prune' and images left behind by exited containers is relevant. The container has an additional runtime layer for any filesystem changes made on-the-fly.

Once #55 is merged, please consider adding common misconceptions or challenging threshold concepts to the the instructor notes or this issue.