I have a question about writing SLURM directives for Python multiprocessing codes to be run on high-performance computing (HPC) platforms. If I want to set the number of processes in the code to 8, should I set ‘ntasks=8’ or ‘cpus-per-task=8’ in the SLURM directive set-up? Some people say that ‘cpus-per-task’ refers to the number of threads, whereas Python multiprocessing does not use multithreading method due to Global Interpreter Lock (GIL).
I am confident that Python multiprocessing definitely works. Because I tried it on my local laptop and I saw a decrease in computing time with the increased number of processes. So, the question is how to set up the SLURM directives correctly in order for codes using multiprocessing to be run on HPC.