container creation failed: mount hook function failure: destination /storage doesn't exist in container about apptainer HOT 6 OPEN

Hi there and thank you for your patience.

I'm not really familiar with this workflow but I think according to the linked docs that you are running the script run_singularity.sh and it is probably failing on the command here:

# execute command in singularity container
singularity exec \
    -B $TMPDIR/docker-isaac-sim/cache/kit:${DOCKER_ISAACSIM_PATH}/kit/cache:rw \
    -B $TMPDIR/docker-isaac-sim/cache/ov:${DOCKER_USER_HOME}/.cache/ov:rw \
    -B $TMPDIR/docker-isaac-sim/cache/pip:${DOCKER_USER_HOME}/.cache/pip:rw \
    -B $TMPDIR/docker-isaac-sim/cache/glcache:${DOCKER_USER_HOME}/.cache/nvidia/GLCache:rw \
    -B $TMPDIR/docker-isaac-sim/cache/computecache:${DOCKER_USER_HOME}/.nv/ComputeCache:rw \
    -B $TMPDIR/docker-isaac-sim/logs:${DOCKER_USER_HOME}/.nvidia-omniverse/logs:rw \
    -B $TMPDIR/docker-isaac-sim/data:${DOCKER_USER_HOME}/.local/share/ov/data:rw \
    -B $TMPDIR/docker-isaac-sim/documents:${DOCKER_USER_HOME}/Documents:rw \
    -B $TMPDIR/orbit:/workspace/orbit:rw \
    -B $CLUSTER_ORBIT_DIR/logs:/workspace/orbit/logs:rw \
    --nv --writable --containall $TMPDIR/orbit.sif \
    bash -c "cd /workspace/orbit && /isaac-sim/python.sh ${CLUSTER_PYTHON_EXECUTABLE} $@"

It looks like that script is setting up the bind-mounts a little earlier in the code.

setup_directories() {
    # Check and create directories
    for dir in \
        "${CLUSTER_ISAAC_SIM_CACHE_DIR}/cache/kit" \
        "${CLUSTER_ISAAC_SIM_CACHE_DIR}/cache/ov" \
        "${CLUSTER_ISAAC_SIM_CACHE_DIR}/cache/pip" \
        "${CLUSTER_ISAAC_SIM_CACHE_DIR}/cache/glcache" \
        "${CLUSTER_ISAAC_SIM_CACHE_DIR}/cache/computecache" \
        "${CLUSTER_ISAAC_SIM_CACHE_DIR}/logs" \
        "${CLUSTER_ISAAC_SIM_CACHE_DIR}/data" \
        "${CLUSTER_ISAAC_SIM_CACHE_DIR}/documents"; do
        if [ ! -d "$dir" ]; then
            mkdir -p "$dir"
            echo "Created directory: $dir"
        fi
    done
}

I think your error is telling you that bind-mounts/NVIDIA bind are incompatible with the writable option because of the way that your environment is configured. You might have to edit that script. Maybe you could use a persistent overlay instead if you need the container to be writable?

You could probably figure out how to execute the same workflow fairly easily with straight Apptainer commands instead of relying on the wrapper scripts. It might be useful to remove this abstraction and use the underlying technology since the abstraction appears to be leaky and the underlying container tech (Apptainer) is pretty user friendly.

from apptainer.

Thank you for your help @GodloveD! I am indeed running the run_singularity.sh script. I do have some follow-up questions.

What about the environment configuration could be incompatible with the bind-mounts? And do you have a clue as to what part would need to be edited? My environment is configured with the following:

# Accept the NVIDIA Omniverse EULA by default
ACCEPT_EULA=Y
# NVIDIA Isaac Sim version to use (e.g. 2022.2.1)
ISAACSIM_VERSION=2023.1.0-hotfix.1
# Derived from the default path in the NVIDIA provided Isaac Sim container
DOCKER_ISAACSIM_PATH=/isaac-sim
# Docker user directory - by default this is the root user's home directory
DOCKER_USER_HOME=/root

###
# Cluster specific settings
###

# Docker cache dir for Isaac Sim (has to end on docker-isaac-sim)
# e.g. /cluster/scratch/$USER/docker-isaac-sim
CLUSTER_ISAAC_SIM_CACHE_DIR=/storage/workspaces/a*****/w****/lars/docker-isaac-sim
# Orbit directory on the cluster (has to end on orbit)
# e.g. /cluster/home/$USER/orbit
CLUSTER_ORBIT_DIR=/storage/homefs/l******/orbit
# Cluster login
CLUSTER_LOGIN=*****@****.ch
# Cluster scratch directory to store the SIF file
# e.g. /cluster/scratch/$USER
CLUSTER_SIF_PATH=/storage/workspaces/a*****/w****/lars
# Python executable within orbit directory to run with the submitted job
CLUSTER_PYTHON_EXECUTABLE=source/standalone/tutorials/00_sim/create_empty.py

Maybe you could use a persistent overlay instead if you need the container to be writable?

As I'm new to apptainer, could you explain why the --writable flag in singularity exec would not suffice here? Should I follow these steps instead?

from apptainer.

I think that the following error in your original post is telling us that Apptainer cannot create the appropriate nested directory structure on the fly within your orbit container because overlay and underlay are not available or misconfigured or something.

WARNING: By using --writable, Apptainer can't create /storage destination automatically without overlay or underlay

This is probably because something is not installed on the host system or apptainer.conf is disallowing this feature. @DrDaveD or @cclerget might know the cause. You'd probably need admin rights on the system to fix that issue.

As for my proposed workaround I would suggest using these docs instead since you are using Apptainer under the hood here.

from apptainer.

I manually added the /storage folder to the orbit.sif folder, now it gives the same error with another mount:

(run_singularity.py): Called on compute node with arguments
WARNING: nv files may not be bound with --writable
WARNING: By using --writable, Apptainer can't create /root/.cache/ov destination automatically without overlay or underlay
FATAL:   container creation failed: mount hook function failure: mount /scratch/local/4319483/docker-isaac-sim/cache/ov->/root/.cache/ov error: while mounting /scratch/local/4319483/docker-isaac-sim/cache/ov: destination /root/.cache/ov doesn't exist in container
(run_singularity.py): Return

However, unlike before, the /root/.cache/ov does exist in the orbit.sif folder, so I can not do the same trick again.

Removing some of the binds gives the same error for another bind, e.g. FATAL: container creation failed: mount hook function failure: mount /scratch/local/4319775/docker-isaac-sim/documents->/root/Documents error: while mounting /scratch/local/4319775/docker-isaac-sim/documents: destination /root/Documents doesn't exist in container, so something seems to be going wrong with the mounting in general.

I will try your proposed workaround (or contact IT if your collaborators know the cause), thank you.

from apptainer.

The --writable option disables overlay and underlay, as discussed in this doc section. In order to be writable with a sif file, you must have a persistent overlay partition embedded in the sif file.

from apptainer.

By the way, version 1.1.3 is quite old. I recommend upgrading to the latest release.

from apptainer.