I have 5 virtual machines (1 client, 1 director, 2 servers). The client and director (in direct routing) are on the same physical machine, while the 2 servers each is on different machines. I am running experiments with tailbench applications. When the client does queries to the linux virtual server director and this forwards the connections to server A, the results are ok. But when the connections are forwarded to server B, there are queries that takes longer, I mean, that sometimes the responses have bigger response times. However, if the client communicates directly with the server B, this variability does not appear. I check all lvs director and servers configurations but I did it in accordance to how to set direct routing.

What I want to try is to build a mini experimental infraestructure to study tail latency with tailbench and load balancing. However, with this problem, the results I get is not reasonable due to the variability that occurs most of the time (making the normal results with 100 queries per second from 3ms to more than 1000ms) and later is stabiliced.

So I want to know what type of problems could caused the huge variability.

Thank you