I have a program that performs a calculation, ran from the command line on a Linux terminal. It uses a command like

program_name -i input_file -o output_file1 -r output_file2 -w dependency1 -d dependency2

I have a wrapper script in Python that I am using to do some file handling and executing the above command. The script is being ran executed from the Linux terminal as well. Since the calculation program can take a long time to run (from minutes to hours), I would like to be able to nohup the terminal output (CPU and program time written to output every 10sec) to a file and send the process to the background so that I can continue with other terminal operations. The command I’m actually trying to execute as a subprocess is

program_name -i input_file -o output_file1 -r output_file2 -w dependency1 -d dependency2 </dev/null >output.log 2>&1 &

The problem is program_name and its dependencies are stored in a shared network location, due to limitations on argument length I can’t run the program using paths to the program/files. I need to create symbolic links to the program/files, run the program, and then unlink the program and files. If I run the subprocess in the terminal foreground, this works fine. However, when I try to run the subprocess in the background, Python won’t wait for it to complete before moving onto the next piece of code, which is unlinking the program and dependencies and subsequently causing the program to fail.

My code works like this

# create symbolic links
for file in [program_name, dependency1, dependency2, dependency3]:
    src = path_to_program / file
    dst = current_directory / file

    try:
        os.symlink(src, dst)
    except FileExistsError:
        pass

# create command
cmd = "program_name -i {inp} -o {out1} -r {out2} -w {dep1} -d {dep2} </dev/null >output.log 2>&1 &".format(
    inp=input_file
    out1=output_file1
    out2=output_file2
    dep1=dependency1
    dep2=dependency2
)         

proc = subprocess.run(cmd, stderr=None, stdout=None, shell=True)

# remove symbolic links
for file in [program_name, dependency1, dependency2, dependency3]:
    os.unlink(current_directory / file)

If the subprocess is ran in foreground by removing the end & the script waits for the subprocess to complete before unlinking the files. If ran in background, as soon as the subprocess is started the script proceeds to unlink the files and break the subprocess.

What am I missing? As far as I understand subprocess.run() should wait for the subprocess to complete before continuing, but it doesn’t appear to be doing so. I’ve also tried using subprocess.Popen() with .wait() and .communicate() ,subprocess.call() ,subprocess.check_call() , and os.system() and got the same behavior with all of them.

If I try to use shell=False like

proc = subprocess.run(shlex.split(cmd), stderr=None, stdout=None, shell=False)

Then program_name tries reading in </dev/null >output.log 2>&1 & as arguments and returns errors.