We are using Locust to run performance tests on our FastAPI endpoints on Google Cloud Platform. Intermittently, we encounter an issue that would not allow us to stop the test due to an error related to Greenlet/Gevent as soon as we click the “Stop” button. Locust will get stuck at the CleanUp stage while the requests are continue being sent.

We find that this error will have a higher chance to occur when we set up larger concurrent peak users and higher spawn rate. It will almost certainly occur if we set 300 peak users and 100 spawn rate.

Also we did not run anything at on_stop()

The error logs:

DEFAULT 2024-05-09T16:01:22.594432Z Exception ignored in: <finalize object at 0x3ef2919c3b20; dead>
ERROR 2024-05-09T16:01:22.595503Z Traceback (most recent call last): File “/usr/local/lib/python3.11/weakref.py”, line 590, in call return info.func(*info.args, **(info.kwargs or {}))
DEFAULT 2024-05-09T16:01:22.595511Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
DEFAULT 2024-05-09T16:01:22.595517Z File “/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py”, line 1133, in _close_pool_connections
DEFAULT 2024-05-09T16:01:22.597778Z conn = pool.get(block=False)
DEFAULT 2024-05-09T16:01:22.597844Z ^^^^^^^^^^^^^^^^^^^^^
DEFAULT 2024-05-09T16:01:22.597855Z File “/usr/local/lib/python3.11/queue.py”, line 182, in get
DEFAULT 2024-05-09T16:01:22.598757Z self.not_full.notify()
DEFAULT 2024-05-09T16:01:22.598854Z File “/usr/local/lib/python3.11/threading.py”, line 369, in notify
DEFAULT 2024-05-09T16:01:22.600847Z if not self._is_owned():
DEFAULT 2024-05-09T16:01:22.600879Z ^^^^^^^^^^^^^^^^
DEFAULT 2024-05-09T16:01:22.600883Z File “/usr/local/lib/python3.11/threading.py”, line 282, in _is_owned
DEFAULT 2024-05-09T16:01:22.601021Z if self._lock.acquire(False):
DEFAULT 2024-05-09T16:01:22.601056Z ^^^^^^^^^^^^^^^^^^^^^^^^^
DEFAULT 2024-05-09T16:01:22.601060Z File “/usr/local/lib/python3.11/site-packages/gevent/thread.py”, line 132, in acquire
DEFAULT 2024-05-09T16:01:22.604530Z sleep()
DEFAULT 2024-05-09T16:01:22.604547Z File “/usr/local/lib/python3.11/site-packages/gevent/hub.py”, line 159, in sleep
DEFAULT 2024-05-09T16:01:22.606630Z waiter.get()
DEFAULT 2024-05-09T16:01:22.606674Z File “src/gevent/_waiter.py”, line 143, in gevent._gevent_c_waiter.Waiter.get
DEFAULT 2024-05-09T16:01:22.606865Z File “src/gevent/_waiter.py”, line 154, in gevent._gevent_c_waiter.Waiter.get
DEFAULT 2024-05-09T16:01:22.606906Z File “src/gevent/_greenlet_primitives.py”, line 61, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch
DEFAULT 2024-05-09T16:01:22.606963Z File “src/gevent/_greenlet_primitives.py”, line 61, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch
DEFAULT 2024-05-09T16:01:22.607023Z File “src/gevent/_greenlet_primitives.py”, line 65, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch
DEFAULT 2024-05-09T16:01:22.607071Z File “src/gevent/_gevent_c_greenlet_primitives.pxd”, line 35, in gevent._gevent_c_greenlet_primitives._greenlet_switch
DEFAULT 2024-05-09T16:01:22.607131Z greenlet.GreenletExit:

We have found this bug issue that looks very similar to our case and therefore have updated locust to 2.17.0, but the issue still persist.
https://github.com/locustio/locust/issues/2401

Tried fixes:

• Update locust to 2.17.0
• Update locust to 2.27.0
• Update gevent to 24.2.1