With python multiprocessing, Let’s assume, I’m creating 10 processes by forking the main process. We know that child process inherit the state of the parent process in its own memory space.
I have a SQLAlchemy engine created in the parent process with pool size of 5. During the time of forking there aren’t any connections in the pool.
Since the child processes have their own memory space that means the engine inherited should be only accessible to them, right?
- If that’s the case, Does it mean the total number of active
connections possible would be 2 [ number of processes ] * 5 [ pool_size ] = 10? - If not, How does the connection pool work here?
import multiprocessing
import os
# Create an engine with a connection pool in the parent process
engine = create_engine('sqlite:///example.db', pool_size=5)
def initializer():
# Connections in the pool are gracefully removed!
engine.dispose(close=False)
def get_stuff():
with engine.connect() as conn:
conn.execute(text("..."))
with multiprocessing.Pool(10,initializer=initializer) as pool:
pool.apply(get_stuff)