My IT department kindly set me up with a Linux VM with Slurm and I’m learning commands. I’m changing numbers a bit, but let’s say there are 2 nodes each with 5 cores/CPUs (I think they’re the same here?). I want to run 6 independent iterations of a function, with 3 subtasks each. The first 2 subtasks (a, b) can be run in parallel, but the final one (c) requires the first 2 to finish. For simplicity we can say each subtask takes 1 minute.

Let’s also say I start set A with iterations 1-6. But then I realize I would like to run set B iterations 7-14 at a higher priority and immediately start that next. In this situation I would like subtasks a and b for 5 of the iterations in set A to complete during minute 1. For minute 2, I would like subtasks a and b for 5 of the iterations in set B to complete. Minute 3 I would like the final 8 subtaks of B to complete along with 2 subtasks in A. Minute 4 would be the final 6 subtasks of A.

Is this possible? How would I go about this?

A MWE is below, but just starting A when I run the top command, I see some CPU usage is 100% for some processes, and 50% for others. With the 50% processes, there are more total running than number of cores. I would prefer the script finishes tasks and waits for a CPU to become free before starting the next because I would think this would be faster to get iterations finished. In this example though, the runtime was always 2 minutes even at 50%, maybe due to the nature of the stress command? All iterations started and ended at the same time too for some reason.

Command I ran
./test_main.sh

Content of test_main.sh

#!/bin/bash

rm -rf 'out'
mkdir -p 'out/'

for i1 in {1..6}; do #Changed to 7..14 for B
 printf -v i_str "%03d" $i1 #add zero-padding
 sbatch ./test_sub.sh "$i_str"
done

Content of test_sub.sh below. I’ve tried playing around with --hint=nomultithread, --cpus-per-task=1, and --ntasks-per-node=5. In the srun command I’ve tried with and without --exclusive.

#!/bin/bash
#
#SBATCH --job-name setA   ## name that will show up in the queue. Change for set B
#SBATCH --output slurm-%j.out   ## filename of the output; the %j is equal to jobID
#SBATCH --nodes=2              ## Number of nodes
#SBATCH --oversubscribe    ## Letting multiple jobs run concurrently
#SBATCH --partition=mypartition     ## the partitions to run in (comma separated). 
#SBATCH --nice=10000  ## Change to a smaller value for set B

in_dir=$1

run_s="$SECONDS" #start a timer

work_dir=$PWD
mkdir -p 'out/'$in_dir
cd 'out/'$in_dir

# Save some tracking info
hostname > example.out #always the name of the primary node. Not important to my task though
echo '('`date`')' >> example.out

srun -N1 -n1 --exclusive "$work_dir"/test_sub2.sh 'alpha' &
srun -N1 -n1 --exclusive "$work_dir"/test_sub2.sh 'beta' &
wait #wait for subprocess alpha and beta to finish

echo '('`date`')' >> example.out

# Can now run gamma
srun -N1 -n1 --exclusive "$work_dir"/test_sub2.sh 'gamma' &
wait

# Save additional tracking info after gamma finishes
echo '('`date`')' >> example.out
loop_rt=$( echo " scale = 2; ( "$SECONDS"- "run_s" ) / 60 " | bc )
echo "$loop_rt" >> example.out  #Runtime

The content of test_sub2.sh

#!/bin/bash

greekLtr=$1

hostname > "$greekLtr".out
echo '('`date`')' >> "$greekLtr".out

stress --cpu 1 --timeout 60

echo '('`date`')' >> "$greekLtr".out

In my slurm.conf file under scheduling I have

# SCHEDULING
SchedulerType=sched/backfill
SelectType=select/cons_tres
SelectTypeParameters=CR_CPU

Pointers to similar questions would also be greatly appreciated! Thanks!