The view where the exception happens is processing a big dictionary with about ~350K entries. The exception is a KeyError.
From what I can see from the CProfile logs, Django is trying to iterate the dictionary to get traceback data.
This is simplified code from the view:
@login_required
def export_plan_data(request):
try:
form = PortfolioPlanningForm(request.GET, user=request.user)
file = get_export_file(user=request.user, form.get_params()) # This is where the exception happens
filename = f"Plan {timezone.now().strftime('%Y-%m-%d %H:%M:%S')}.xlsx"
response = HttpResponse(file, content_type="application/vnd.openxmlformats-officedocument.spreadsheetml.sheet")
response["Content-Disposition"] = f"attachment; filename={filename}"
return response
except Exception as e:
return JsonResponse({"error": "An error occurred while exporting the data"}, status=500)
If I surround the entire view in a try/except block and return my custom response when an exception occurs it returns in about 10 seconds, but if I remove the try/except block and let Django handle the exception it takes about 5 minutes.
Here’s the Cprofile dump:
292357310 function calls (260422005 primitive calls) in 247.585 seconds
Ordered by: cumulative time
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 247.585 247.585 /usr/local/lib/python3.9/site-packages/django/core/handlers/exception.py:44(inner)
1 0.000 0.000 241.196 241.196 /usr/local/lib/python3.9/site-packages/django/core/handlers/exception.py:54(response_for_exception)
1 0.000 0.000 241.194 241.194 /usr/local/lib/python3.9/site-packages/django/core/handlers/exception.py:140(handle_uncaught_exception)
1 0.000 0.000 241.194 241.194 /usr/local/lib/python3.9/site-packages/django/views/debug.py:50(technical_500_response)
1 0.000 0.000 241.192 241.192 /usr/local/lib/python3.9/site-packages/django/views/debug.py:341(get_traceback_html)
429 0.001 0.000 241.047 0.562 /usr/local/lib/python3.9/site-packages/django/template/defaultfilters.py:916(pprint)
429 0.001 0.000 241.046 0.562 /usr/local/lib/python3.9/pprint.py:55(pformat)
429 0.001 0.000 241.044 0.562 /usr/local/lib/python3.9/pprint.py:151(pformat)
1 0.001 0.001 241.040 241.040 /usr/local/lib/python3.9/site-packages/django/views/debug.py:269(get_traceback_data)
3228318/429 12.640 0.000 240.785 0.561 /usr/local/lib/python3.9/pprint.py:163(_format)
721108/16 2.370 0.000 239.680 14.980 /usr/local/lib/python3.9/pprint.py:189(_pprint_dict)
721108/16 11.289 0.000 239.680 14.980 /usr/local/lib/python3.9/pprint.py:372(_format_dict_items)
2/1 0.002 0.001 239.667 239.667 /usr/local/lib/python3.9/pprint.py:446(_pprint_default_dict)
187641/61263 0.431 0.000 205.978 0.003 /usr/local/lib/python3.9/pprint.py:219(_pprint_list)
187647/61269 1.401 0.000 205.810 0.003 /usr/local/lib/python3.9/pprint.py:389(_format_items)
6187933 9.569 0.000 158.701 0.000 /usr/local/lib/python3.9/pprint.py:430(_repr)
6187933 6.127 0.000 147.533 0.000 /usr/local/lib/python3.9/pprint.py:439(format)
33189244/6187933 71.383 0.000 141.406 0.000 /usr/local/lib/python3.9/pprint.py:529(_safe_repr)
5405048 30.852 0.000 70.697 0.000 {built-in method builtins.sorted}
1635495 22.010 0.000 35.767 0.000 /usr/local/lib/python3.9/pprint.py:256(_pprint_str)
15906637 27.737 0.000 33.766 0.000 /usr/local/lib/python3.9/pprint.py:99(_safe_tuple)
41216386/41216385 10.506 0.000 10.536 0.000 {built-in method builtins.repr}
1635719 2.060 0.000 7.389 0.000 /usr/local/lib/python3.9/re.py:233(findall)
1 0.000 0.000 6.389 6.389 /usr/local/lib/python3.9/site-packages/django/core/handlers/base.py:160(_get_response)
1 0.000 0.000 6.389 6.389 /usr/local/lib/python3.9/site-packages/django/contrib/auth/decorators.py:18(_wrapped_view)
1 0.000 0.000 6.373 6.373 /home/app/webapp/apps/home/views/portfolio.py:733(export_plan_data)
1 0.000 0.000 6.358 6.358 /home/app/webapp/apps/home/views/portfolio.py:744(<lambda>)
1 0.001 0.001 6.358 6.358 /home/app/webapp/apps/home/usecases/portfolio/export.py:129(get_export_file)
1 0.098 0.098 6.347 6.347 /home/app/webapp/apps/home/usecases/portfolio/export.py:76(get_data_for_detail_export)
1 0.001 0.001 6.235 6.235 /home/app/webapp/apps/home/usecases/portfolio/get_dashboard_data.py:186(get_fees_for_chart)
1 0.000 0.000 6.224 6.224 /home/app/webapp/apps/home/usecases/portfolio/get_dashboard_data.py:60(get_dashboard_data)
1 0.684 0.684 6.221 6.221 /home/app/webapp/apps/home/usecases/portfolio/get_dashboard_data.py:37(get_enriched_fees_from_portfolio_items)
19596401 6.078 0.000 6.078 0.000 /usr/local/lib/python3.9/pprint.py:92(__lt__)
31813274 6.028 0.000 6.028 0.000 /usr/local/lib/python3.9/pprint.py:89(__init__)
14 0.037 0.003 5.071 0.362 /usr/local/lib/python3.9/site-packages/django/db/models/query.py:1322(_fetch_all)
4 0.000 0.000 5.015 1.254 /usr/local/lib/python3.9/site-packages/django/db/models/query.py:265(__iter__)
61235 0.115 0.000 4.969 0.000 /usr/local/lib/python3.9/site-packages/django/db/models/query.py:97(__iter__)
15 0.000 0.000 4.037 0.269 /usr/local/lib/python3.9/site-packages/silk/sql.py:64(execute_sql)
10 0.001 0.000 3.985 0.399 /usr/local/lib/python3.9/site-packages/django/db/models/sql/compiler.py:1147(execute_sql)
9 0.000 0.000 3.970 0.441 /usr/local/lib/python3.9/site-packages/django/db/models/sql/compiler.py:1126(results_iter)
20 0.000 0.000 3.746 0.187 /usr/local/lib/python3.9/site-packages/django/db/backends/utils.py:96(execute)
20 0.000 0.000 3.745 0.187 /usr/local/lib/python3.9/site-packages/django/db/backends/utils.py:65(execute)
20 0.000 0.000 3.745 0.187 /usr/local/lib/python3.9/site-packages/django/db/backends/utils.py:71(_execute_with_wrappers)
20 0.000 0.000 3.745 0.187 /usr/local/lib/python3.9/site-packages/django/db/backends/utils.py:77(_execute)
20 3.744 0.187 3.744 0.187 {method 'execute' of 'psycopg2.extensions.cursor' objects}
1635719 3.186 0.000 3.186 0.000 {method 'findall' of 're.Pattern' objects}
20916708 2.776 0.000 2.776 0.000 {method 'write' of '_io.StringIO' objects}
28180497/28179681 2.571 0.000 2.590 0.000 {built-in method builtins.len}
1635771 1.448 0.000 2.151 0.000 /usr/local/lib/python3.9/re.py:289(_compile)
20109045 1.950 0.000 1.950 0.000 {method 'append' of 'list' objects}
6188513 1.599 0.000 1.599 0.000 {method 'copy' of 'dict' objects}
82638 0.184 0.000 1.106 0.000 /usr/local/lib/python3.9/json/__init__.py:299(loads)
8791283 1.100 0.000 1.100 0.000 {built-in method builtins.id}
5564885/5564848 1.033 0.000 1.036 0.000 {method 'join' of 'str' objects}
6242997/6242534 0.963 0.000 0.972 0.000 {built-in method builtins.getattr}
9339835 0.917 0.000 0.917 0.000 {built-in method builtins.issubclass}
82638 0.195 0.000 0.901 0.000 /usr/local/lib/python3.9/json/decoder.py:332(decode)
61239 0.123 0.000 0.764 0.000 /usr/local/lib/python3.9/site-packages/django/db/models/sql/compiler.py:1115(apply_converters)
1884024/1884009 0.738 0.000 0.739 0.000 {built-in method builtins.isinstance}
61234 0.044 0.000 0.641 0.000 /usr/local/lib/python3.9/site-packages/django/db/models/fields/json.py:75(from_db_value)
5405343 0.631 0.000 0.631 0.000 {method 'items' of 'dict' objects}
82638 0.605 0.000 0.605 0.000 /usr/local/lib/python3.9/json/decoder.py:343(raw_decode)
1635522 0.426 0.000 0.426 0.000 {method 'splitlines' of 'str' objects}
2777194 0.422 0.000 0.422 0.000 {method 'get' of 'dict' objects}
426 0.257 0.001 0.257 0.001 {method 'getvalue' of '_io.StringIO' objects}
620 0.001 0.000 0.231 0.000 /usr/local/lib/python3.9/site-packages/django/db/models/sql/compiler.py:1640(cursor_iter)
620 0.002 0.000 0.228 0.000 /usr/local/lib/python3.9/site-packages/django/db/models/sql/compiler.py:1646(<lambda>)
626 0.002 0.000 0.225 0.000 /usr/local/lib/python3.9/site-packages/django/db/utils.py:95(inner)
1636531 0.223 0.000 0.223 0.000 {method 'pop' of 'list' objects}
620 0.149 0.000 0.223 0.000 {method 'fetchmany' of 'psycopg2.extensions.cursor' objects}
758842 0.168 0.000 0.168 0.000 {method 'startswith' of 'str' objects}
455971 0.158 0.000 0.159 0.000 {built-in method builtins.next}
1 0.000 0.000 0.127 0.127 /usr/local/lib/python3.9/site-packages/django/template/base.py:164(render)
1 0.000 0.000 0.127 0.127 /usr/local/lib/python3.9/site-packages/django/template/base.py:161(_render)
341/1 0.004 0.000 0.127 0.127 /usr/local/lib/python3.9/site-packages/django/template/base.py:934(render)
6829/47 0.006 0.000 0.127 0.003 /usr/local/lib/python3.9/site-packages/django/template/base.py:897(render_annotated)
61234 0.122 0.000 0.122 0.000 /usr/local/lib/python3.9/site-packages/django/db/models/query.py:110(<dictcomp>)
87/6 0.010 0.000 0.121 0.020 /usr/local/lib/python3.9/site-packages/django/template/defaulttags.py:157(render)
2448 0.005 0.000 0.096 0.000 /usr/local/lib/python3.9/site-packages/django/template/base.py:986(render)
312/16 0.001 0.000 0.084 0.005 /usr/local/lib/python3.9/site-packages/django/template/defaulttags.py:300(render)
165406 0.076 0.000 0.076 0.000 {method 'match' of 're.Pattern' objects}
122468 0.065 0.000 0.074 0.000 /usr/local/lib/python3.9/site-packages/psycopg2/_json.py:159(typecast_json)
14 0.000 0.000 0.064 0.005 /usr/local/lib/python3.9/site-packages/django/db/models/query.py:45(__iter__)
2956 0.006 0.000 0.053 0.000 /usr/local/lib/python3.9/site-packages/django/template/base.py:668(resolve)
7 0.000 0.000 0.048 0.007 /usr/local/lib/python3.9/site-packages/django/db/models/query.py:261(__len__)
2448 0.008 0.000 0.041 0.000 /usr/local/lib/python3.9/site-packages/django/template/base.py:963(render_value_in_context)
25 0.001 0.000 0.038 0.002 /usr/local/lib/python3.9/site-packages/django/db/models/sql/compiler.py:503(as_sql)
187696 0.033 0.000 0.033 0.000 {built-in method builtins.iter}
1212/1211 0.001 0.000 0.032 0.000 /usr/local/lib/python3.9/site-packages/django/utils/functional.py:244(inner)