I have implemented a circular queue using an array in C++. The queue uses two atomic variables, front and rear, to manage enqueue and dequeue operations. When an element is enqueued, rear is incremented, and when an element is dequeued, front is incremented.

I have created 16 instances of this circular queue, each with a size of 2^23 elements.

There are two scenarios I’m testing:

Scenario 1:

There are 2 reader threads.
Each reader thread is pinned to a separate CPU core and isolated.
Each reader is responsible for 8 of the 16 queues and sequentially checks each queue for packets, performing dequeue operations when packets are found, followed by certain processing.
Another thread enqueues elements into different circular queues.

Scenario 2:

There are 8 reader threads.
Each reader thread is pinned to a separate CPU core and isolated.
Each reader is responsible for 2 of the 16 queues and sequentially checks each queue for packets, performing dequeue operations when packets are found, followed by certain processing.
Another thread enqueues elements into different circular queues.

Metric:

The primary metric for comparison in both scenarios is the time taken by an element between enqueue and dequeue (latency or time spent by a packet inside the queue).

Problem:

Naturally, I expected that increasing the number of reader threads would decrease the latency. However, I observed the opposite: the latency increased when more readers were employed.

Additionally, I noticed that when more reader threads are employed, some of the threads have lower CPU utilization (around 70-80%) despite having an infinite while loop present in them.

Question:

Why might the latency increase when more readers are added? Are there specific concurrency or cache-related issues in my implementation that could explain this behavior?

What could be causing the lower CPU utilization in some threads when more reader threads are used?

Additional Information:
The code is written in C++.
Core pinning and isolation have been done for all threads to avoid interference.

I have tried checking for any contention issues or any other but could,nt find any.