I have a django application that I would like to host on my apache server. Use case of the application is to receive requests and performs compute intensive operations over a set of given inputs. To parallelise my computations I used future.ProcessPool executor in the core of the django app. Parallelisation in indispensable for my application.

While the above setup works locally, I face BrokenProcessPool error especially when the app is hosted on the apache server. Tried few tricks in some related questions (here) but I have no clear idea what is happening or how to resolve it. Could someone please help me understand:

1. Technically, what is the underlying issue in my setup?
2. How do I resolve it without massively changing my existing framework(Django, Apache)?
3. Can you provide me references for an ideal workflow used in some well known data-science or graphics web applications using Django and parallel computations. This could help me revise my current framework.

Below is some relevant information

• views.py looks like

  Plain text

  Copy to clipboard

  Open code in new window

  EnlighterJS 3 Syntax Highlighter

  <code> import core.ray_trace

  def get_heatmap(request):

  list_of_inputs = request.POST.get('inputs')

  # parallel computation of list happens inside raytrace

  response_data = ray_trace(list_of_inputs)

  return JsonResponse(response_data, status=200)

  </code>

  <code> import core.ray_trace def get_heatmap(request): list_of_inputs = request.POST.get('inputs') # parallel computation of list happens inside raytrace response_data = ray_trace(list_of_inputs) return JsonResponse(response_data, status=200) </code>

   import core.ray_trace
  
    def get_heatmap(request):
       list_of_inputs = request.POST.get('inputs')
        # parallel computation of list happens inside raytrace
        response_data = ray_trace(list_of_inputs)
        return JsonResponse(response_data, status=200)
  

• Apache config for my project

  Plain text

  Copy to clipboard

  Open code in new window

  EnlighterJS 3 Syntax Highlighter

  <code> WSGIDaemonProcess sp python-path=/home/ubuntu/_/spaceplanning python-home=/home/ubuntu/_/venv processes=3 threads=10 request-timeout=600

  WSGIProcessGroup sp

  WSGIScriptAlias / /home/ubuntu/_/sp/spAPI/wsgi.py

  </code>

  <code> WSGIDaemonProcess sp python-path=/home/ubuntu/_/spaceplanning python-home=/home/ubuntu/_/venv processes=3 threads=10 request-timeout=600 WSGIProcessGroup sp WSGIScriptAlias / /home/ubuntu/_/sp/spAPI/wsgi.py </code>

     WSGIDaemonProcess sp python-path=/home/ubuntu/_/spaceplanning python-home=/home/ubuntu/_/venv processes=3 threads=10 request-timeout=600
     WSGIProcessGroup sp
     WSGIScriptAlias / /home/ubuntu/_/sp/spAPI/wsgi.py
  

• ps aux | grep apache

• Error message and Traceback: concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.

I read switching my frame work to Celery or gunicorn server could help but honestly I could not catch on the intricates of the details provided.

Finally, are there any good web hosting services catering for data science and graphics applications that I could just use to unburden myself from hosting?