I have a Django app hosted on Heroku. Some user requests to this app require making long-running queries to external resources. In order to get around the Heroku 30 request timeout, I created a system where these long running queries are given unique IDs and handed to a Python ThreadPoolExecutor so as not to hold off the query. The original query returns immediately and the font end JS code periodically checks in for a results.

This system is working great but since deploying that change I routinely get “FATAL: too many connections” errors from my app. It appears as though the automatic connection management Django provides doesn’t work when ThreadPoolExecutors are in use.

Unfortunately I have to have access to the DB from the spawned thread in order to save the results of the long-running query. Any recommendation as to how I can fix this so DB connections are managed correctly?