pytorch training issue: insufficient shared memory about ffdl HOT 2 CLOSED

Hi @Eric-Zhang1990, technically you can package your code with the bash zip command if you have multi-directories. (e.g. zip -r model.zip maskrcnn-benchmark)

In the backend, FfDL will unzip the model.zip to the learner container's work_dir. Then, you can modify the command in manifest.yml to execute your workload.
(e.g. python maskrcnn_benchmark/tools/train_net.py)

from ffdl.

@Tomcli Thank you for your kind reply. I loss a parameter '-r' when zip model files.
However, I encounter another problem about 'shm', which is "ERROR: Unexpected bus error encountered in worker. This might be caused by insufficient shared memory (shm).".
my config is:
gpus: 1
cpus: 4
learners: 1
memory: 16Gb

framework:
name: custom
version: "pytorch:1.0-gpu"
command: >
export NGPUS=1; source /etc/profile; bash ${MODEL_DIR}/setup.sh; . env_file; python ${MODEL_DIR}/setup.py build develop; python -m torch.distributed.launch --nproc_per_node=$NUM_GPUS ${MODEL_DIR}/train_net.py;

Thank you.

from ffdl.