We have a Java EJB application hosted on Wildfly 15 which has a lightweight Web API service hosted on Jetty 9 in front of it.
The Web API service just deserialises json requests from the clients and calls the appropriate Remote EJB method; then receives the response from the EJB and creates a JSON response to the client.
We have horizontally scaled a lot – 10 Wildflies and 6 Web API Jetties.
In front of the Jetties we have a HAProxy round-robin balancing the requests towards the jetties.
When we receive a significant load on the system, the HAProxy reports response times from the Jetties of around 4 seconds.
However, a New Relic agent deployed on the Web API jetties reports response time of around 800ms.
The New Relic agent deployed on the EJB app Wildflies reports response times of around 400ms.
Our MySQL DB says the lock wait time is around 150ms.