We have a Java EJB application hosted on Wildfly 15 which has a lightweight Web API service hosted on Jetty 9 in front of it.
The Web API service just deserialises json requests from the clients and calls the appropriate Remote EJB method; then receives the response from the EJB and creates a JSON response to the client.
We have horizontally scaled a lot – 10 Wildflies and 6 Web API Jetties.
In front of the Jetties we have a HAProxy round-robin balancing the requests towards the jetties.
When we receive a significant load on the system, the HAProxy reports response times from the Jetties of around 4 seconds.
However, a New Relic agent deployed on the Web API jetties reports response time of around 800ms.
The New Relic agent deployed on the EJB app Wildflies reports response times of around 400ms.
Our MySQL DB says the lock wait time is around 150ms.

I would like to ask for help in understanding where can the discrepancy of the response times between the HAProxy and the NewRelic agents come from. I would understand a small discrepancy of 100-200ms, but not 5X.

We tried horizontally scaling, but it didn’t help.
While all this is happening, both the Wildflies and the Jetties are not struggling at all: Jetties are around 10% CPU load, and Wildflies around 20%.

Thank you.