Global Interpreter Lock (GIL) in Python can limit the true concurrency achieved with multithreading.
Python’s global interpreter lock (“GIL”) prevents multiple threads from executing Python code at the same time. The GIL is an obstacle to using multi-core CPUs from Python efficiently, which is a major obstacle to concurrency. (https://peps.python.org/pep-0703)

I have a script that sends GET requests. I want to speed up this process by sending multiple requests simultaneously by using multi-threading. I tried parallel processing, but it’s much slower than multi-threading.

Below are the runtimes: (I need to do this at a much larger scale, just using 5 as an example)

1. 5 GET requests using a simple sequential for-loop: 14 seconds
2. 5 GET requests using multi-threading with 1 thread: 11 seconds
3. 5 GET requests using multi-threading with 2 thread: 7 seconds
4. 5 GET requests using multi-threading with 15 threads: 3.5 seconds

This all works fine and fast. I’m happy with the speed of the multi-threading. I’m using concurrent.futures.ThreadPoolExecutor(max_workers=15)

Now the issue occurs when I try to implement and combine caching with the concurrent requests.
For single requests, sequential for-loops or multi-threading with 1 core it works fine.
A simple for-loop with caching has a runtime of 2.5 seconds (instead of 14s)
A multi-thread run with 1 thread has a speed of 1 second (instead of 11 seconds)

The problem occurs when I try to run it with more than 1 core, from 2 cores onward, WITH caching enabled. This is obviously due to the same 2 or 10 cores trying to access the same caching resource.
I tried both local and in-process memory caching.

So each core tries to read & write to the same cache file simultaneously.
Apparently requests.get is thread-safe for reading, but not for writing.
So my idea was to maybe delay the writing to the cache file after the multi-threaded loop has finished, but that would be quite some under the hood programming.

Does anyone know the best approach to tackle this issue?

I tried parallel processing – ProcessPoolExecutor (executor.map), but the speed was a let down, it was equal to the 1-core multi-threaded speed of 11s. And I also run into
This would also use only 1 core, but then multiple processes within that 1 core.
But I have to be honest, I may not be implementing the multi-processing correctly, as I run into the following error in some attempts: BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.

Thanks a lot in advance for your help! 🙂