each job starts a virtual machine about cyclecloud-slurm HOT 10 CLOSED

This will depend on what VM type you've selected and how you're submitting your jobs. Job resource requests are taken into account when allocating new nodes.

from cyclecloud-slurm.

I'm using this repo and cyclecloud 7.8.
Currently configured instances are HB60rs and H16r and NCxx_v3.

I'll look into this:
https://groups.google.com/forum/#!searchin/slurm-users/cloud%7Csort:date/slurm-users/qsQiX9MNk7M/Ir-XlN2bBwAJ

from cyclecloud-slurm.

What command are you using to submit your job and are there any #SBATCH lines in your jobs script?

from cyclecloud-slurm.

I was running
a)

#srun hostname &
#srun hostname &
#srun hostname &
#srun hostname &

b) sbatch test-job.sh

$ sinfo
PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
gpu          up   infinite      0    n/a
hpc*         up   infinite      0    n/a
htc          up   infinite      0    n/a

$ sbatch ./test-job.sh
Submitted batch job 4

$ sbatch ./test-job.sh
Submitted batch job 5

 $ sinfo
PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
gpu          up   infinite      0    n/a
hpc*         up   infinite      2 alloc# hpc-pg0-[1-2]
htc          up   infinite      0    n/a

The script is:

$ cat test-job.sh
#!/usr/bin/env bash
#SBATCH -J test-worker
#SBATCH -p hpc
#SBATCH -A WAL
#SBATCH -n 1
#SBATCH --cpus-per-task=1
#SBATCH --mem=1G
#SBATCH -t 240:00:00
JOB_ID=${SLURM_JOB_ID%;*}

export MKL_NUM_THREADS=1
export OPENBLAS_NUM_THREADS=1
export OMP_NUM_THREADS=1

srun hostname

from cyclecloud-slurm.

@anhoward Any issues with my setup, what do you think?

from cyclecloud-slurm.

@tbugfinder no issues that I see there, but I can replicate the issue on my test cluster. Haven't had a chance to dig in much beyond that but I will update with anything I can find.

from cyclecloud-slurm.

I think this is a setting in slurm.conf - I might find it again.

from cyclecloud-slurm.

SelectType=select/linear

from cyclecloud-slurm.

e.g. onprem we've setup SelectType=select/cons_res

from cyclecloud-slurm.

I'm using a different selecttype now utilizing my slurm master configuration.

from cyclecloud-slurm.