I have some trouble with using the multiprocessing.Process. Briefly, I create multiple Process p and start them by calling p.start().

import multiprocessing
def _start_process(args):
    # some other codes to generate some_args
    q = multiprocessing.Queue()
    p = multiprocessing.Process(target=some_func, args=[q, some_args])
    p.start()

# retrieve result from queue, join and close the process in some other functions. 

This works properly in most cases, but I encounter a strange error occasionally when calling start:

Traceback (most recent call last):
  File ".../__init__.py", line 63, in _start_process
    p.start()
  File ".../lib/python3.10/multiprocessing/process.py", line 120, in start
    _cleanup()
  File ".../lib/python3.10/multiprocessing/process.py", line 64, in _cleanup
    if p._popen.poll() is not None:
AttributeError: 'NoneType' object has no attribute 'poll'

The error does not frequently occur. However, I still have to fix this because whenever I run a big job, a single process that does not start will eventually ruin the final results. As this occurs randomly, I could not reliably replicate the error. Such error seems to occur more frequently when I use a large chunk of memory in the main process (~20GB) and start many short processes simultaneously (~40). It is not a pickling error since I only distribute a fraction of my data to each process. I am using RockyLinux 9.0 and python 3.10.

It would be great if someone could provide me clues at what scenario p._popen is None when calling p.start.