I’m attempting to run multiple Python scripts sequentially on different GPUs through a Bash script. My goal is for each GPU to run a series of Python scripts one after the other without overlapping, and each script is specific to that GPU. However, I’m facing an issue where all scripts seem to execute on GPU 0 despite specifying CUDA_VISIBLE_DEVICES.

I tested few different scripts, but all of them located the data to GPU 0:
script 1

#!/bin/bash

# GPU 1
CUDA_VISIBLE_DEVICES=1 bash -c "
python script1.py --params;
python script2.py --params;
...
"     &

# GPU 2
CUDA_VISIBLE_DEVICES=2 bash -c "
python script1.py --params;
python script2.py --params;
...
" &

wait

script 2

#!/bin/bash

# GPU 1
{
CUDA_VISIBLE_DEVICES=1 python script1.py --params
CUDA_VISIBLE_DEVICES=1 python script2.py --params
...
}&

# GPU 2
{
CUDA_VISIBLE_DEVICES=2 python script1.py --params
CUDA_VISIBLE_DEVICES=2 python script2.py --params
}
...

Despite using CUDA_VISIBLE_DEVICES to allocate GPUs, all processes seem to run on GPU 0. I’ve confirmed that my CUDA installation recognizes multiple GPUs with nvidia-smi.

Could someone please explain what might be wrong with my approach or how I can ensure that each script runs on the correct GPU as specified? Is there a better way to structure these commands to enforce the GPU assignments?