For background I am working on a script to train multiple pytorch models. I have a training script that I want to be able to run as a sub process in a gnome terminal. The main reason for this is so I can keep an eye on the training progress. In cases where I might have multiple GPUs I would like to run my training script multiple times in separate windows. To accomplish this I have been using popen. The following code works to open a new terminal window and start the training script

#create a list of commands
commands = []
kd_cfg = KDConfig.read(kd_cfg_path)
cmd = "python scripts/train_torch.py "
for specialist in kd_cfg.specialists:
    cmd += f"--config {kd_cfg.runtime_dict['specialists'][specialist]['config']} "
    ...

# Run each command in a new terminal and store the process object
num_gpus = len(gpus)
free_gpus = copy.deepcopy(gpus)
processes = []
worker_sema = threading.Semaphore(num_gpus)
commands_done = [False for _ in range(len(commands))]

#start the watchdog
watch = threading.Thread(target=watch_dog, args=(processes,free_gpus,commands_done,worker_sema))
watch.start()

for cmd_idx, command in enumerate(commands):

    worker_sema.acquire()

    gpu = free_gpus.pop()
    command += f" --gpu {gpu}" #allocate a free GPU from the list
    split_cmd_arr = shlex.split(command)
    proc = subprocess.Popen(['gnome-terminal', '--'] + split_cmd_arr)

    processes.append( (cmd_idx,gpu,proc) )

The part that I am stuck on is the concurrency control. In order to protect the GPU resource I use a semaphore. My plan was to monitor the process that starts the GNOME terminal and when it has finished release the semaphore to start the next training process. Instead what happens is that all commands are run at the same time. When I test with two commands and limit onto one gpu I still see two terminals open and two trainings will begin. In my watchdog thread code below I see that both the processes are zombies and have no children even WHILE I am watching the training loop execute inside of both terminals without crashing.

   # Check if processes are still running
    while not all(commands_done):
        for cmd_idx, gpu, proc in processes:
            # try:
            # Check if process is still running
            ps_proc = psutil.Process(proc.pid)

            #BC we call bash python out of the gate it executes as a child proc
            ps_proc_children = get_child_processes(proc.pid)
            proc_has_running_children = any(child.is_running for child in ps_proc_children)

            print(f"status: {ps_proc.status()}")
            print(f"children:  {ps_proc_children}")
            if proc_has_running_children:
                print(f"Process {proc.pid} on GPU {gpu} is still running", end='r')
            else:
                print(f"Process {proc.pid} has terminated")
                free_gpus.append(gpu)
                commands_done[cmd_idx] = True
                processes.remove((cmd_idx, gpu, proc))

                ps_proc.wait()
                print(f"removed proc {ps_proc.pid}")
                worker_sema.release()

I thought maybe the subprocess basically starts another process then immediately returns but I am surprised to see that there are no children either. If anyone has any insights they would be much apprecaited.

If it helps this is some example output from the watchdog.

status: zombie
children:  []
Process 4076 has terminated
removed proc 4076
status: zombie
children:  []
Process 4133 has terminated
removed proc 4133