WSO2 APIM Server : 4.1.0
OpenJDK : 64-Bit Server VM (11.0.17+8-LTS mixed mode, sharing)
Linux : RHEL 8.x
CPU/Processor : 4
Memory : 8GB
WSO2 APIM 4.1.0 running on OpenJDK 11 continues to shows 100% more CPU utilization when being used in an environment with average throughput. CPU usage will remain in more than 100% figures and the issue will be resolved after restarting the server. After running for couple of days, the problem is re occurring and only a restart is the work around now.
Thread Usage shows HTTP-Sender I/O dispatcher threads consuming more CPU and and all other threads are showing normal figures.
%CPU CPU NI S TIME PID TID
65.3 – 0 R 98-06:31:07 1786915 1787000 (Decimal -> 1b4478)
78.9 – 0 R 118-14:04:13 1786915 1787001 (Decimal -> 1b4479)
1.0 – 0 R 1-13:32:03 1786915 1787002 (Decimal -> 1b447a)
3.9 – 0 R 5-21:18:34 1786915 1787003 (Decimal -> 1b447b)
1.7 – 0 S 2-15:00:19 1786915 1787200 (Decimal -> 1b4540)
1.5 – 0 S 2-09:38:23 1786915 1786922 (Decimal -> 1b442a)
Thread dump for the HTTP sender threads is given below,
HTTP-Sender I/O dispatcher-1" #68 prio=5 os_prio=0 cpu=**8490642866.44ms** elapsed=12983070.78s tid=0x00007f325801c800 nid=0x**1b4478** runnable [0x00007f3241dc7000]
java.lang.Thread.State: RUNNABLE
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
at java.lang.Thread.run([email protected]/Thread.java:829)
"HTTP-Sender I/O dispatcher-2" #69 prio=5 os_prio=0 cpu=**10245828487.59ms** elapsed=12983070.78s tid=0x00007f325835c000 nid=0x**1b4479** runnable [0x00007f3241cc7000]
java.lang.Thread.State: RUNNABLE
at java.lang.Throwable.fillInStackTrace([email protected]/Native Method)
at java.lang.Throwable.fillInStackTrace([email protected]/Throwable.java:787)
- **locked** <0x00000000f28b12b8> (a org.apache.http.MalformedChunkCodingException)
at java.lang.Throwable.<init>([email protected]/Throwable.java:270)
at java.lang.Exception.<init>([email protected]/Exception.java:66)
at java.io.IOException.<init>([email protected]/IOException.java:58)
at org.apache.http.MalformedChunkCodingException.<init>(MalformedChunkCodingException.java:54)
at org.apache.http.impl.nio.codecs.ChunkDecoder.readChunkHead(ChunkDecoder.java:112)
at org.apache.http.impl.nio.codecs.ChunkDecoder.read(ChunkDecoder.java:205)
at org.apache.synapse.transport.passthru.Pipe.produce(Pipe.java:250)
at org.apache.synapse.transport.passthru.TargetResponse.read(TargetResponse.java:164)
at org.apache.synapse.transport.passthru.TargetHandler.inputReady(TargetHandler.java:606)
at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:265)
at org.apache.synapse.transport.http.conn.LoggingNHttpClientConnection.consumeInput(LoggingNHttpClientConnection.java:115)
at org.apache.synapse.transport.passthru.ClientIODispatch.onInputReady(ClientIODispatch.java:83)
at org.apache.synapse.transport.passthru.ClientIODispatch.onInputReady(ClientIODispatch.java:41)
at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:114)
at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
at java.lang.Thread.run([email protected]/Thread.java:829)
"HTTP-Sender I/O dispatcher-3" #70 prio=5 os_prio=0 cpu=**135099517.41ms** elapsed=12983070.78s tid=0x00007f3258230000 nid=0x1b447a runnable [0x00007f3241bc6000]
java.lang.Thread.State: RUNNABLE
at org.apache.http.impl.nio.codecs.ChunkDecoder.read(ChunkDecoder.java:205)
at org.apache.synapse.transport.passthru.Pipe.produce(Pipe.java:250)
at org.apache.synapse.transport.passthru.TargetResponse.read(TargetResponse.java:164)
at org.apache.synapse.transport.passthru.TargetHandler.inputReady(TargetHandler.java:606)
at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:265)
at org.apache.synapse.transport.http.conn.LoggingNHttpClientConnection.consumeInput(LoggingNHttpClientConnection.java:115)
at org.apache.synapse.transport.passthru.ClientIODispatch.onInputReady(ClientIODispatch.java:83)
at org.apache.synapse.transport.passthru.ClientIODispatch.onInputReady(ClientIODispatch.java:41)
at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:114)
at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
at java.lang.Thread.run([email protected]/Thread.java:829)
"HTTP-Sender I/O dispatcher-4" #71 prio=5 os_prio=0 cpu=**508690354.93ms** elapsed=12983070.78s tid=0x00007f3258231000 nid=0x1b447b runnable [0x00007f3241ac5000]
java.lang.Thread.State: RUNNABLE
at java.lang.Throwable.fillInStackTrace([email protected]/Native Method)
at java.lang.Throwable.fillInStackTrace([email protected]/Throwable.java:787)
- **locked** <0x00000000f244f4b0> (a org.apache.http.MalformedChunkCodingException)
at java.lang.Throwable.<init>([email protected]/Throwable.java:270)
at java.lang.Exception.<init>([email protected]/Exception.java:66)
at java.io.IOException.<init>([email protected]/IOException.java:58)
at org.apache.http.MalformedChunkCodingException.<init>(MalformedChunkCodingException.java:54)
at org.apache.http.impl.nio.codecs.ChunkDecoder.readChunkHead(ChunkDecoder.java:112)
at org.apache.http.impl.nio.codecs.ChunkDecoder.read(ChunkDecoder.java:205)
at org.apache.synapse.transport.passthru.Pipe.produce(Pipe.java:250)
at org.apache.synapse.transport.passthru.TargetResponse.read(TargetResponse.java:164)
at org.apache.synapse.transport.passthru.TargetHandler.inputReady(TargetHandler.java:606)
at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:265)
at org.apache.synapse.transport.http.conn.LoggingNHttpClientConnection.consumeInput(LoggingNHttpClientConnection.java:115)
at org.apache.synapse.transport.passthru.ClientIODispatch.onInputReady(ClientIODispatch.java:83)
at org.apache.synapse.transport.passthru.ClientIODispatch.onInputReady(ClientIODispatch.java:41)
at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:114)
at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
at java.lang.Thread.run([email protected]/Thread.java:829)
Looking into any other metrics or logs can help to root cause the problem. Please assist.
We have not applied any much performance tuning in the system, there are only 10 API definitions and only two applications subscribing to the same. There is a load balancer placed in-front (Apache Web Server) and the load is distributed between two WSO2 APIM servers. All the API definitions uses the OKTA key Manager with introspect enabled for JWT validation. The connections to the OKTA servers (public internet traffic) are directed through an internal proxy server (running on HTTP). Connections to the back end services are plain HTTP. There are multiple instances of back end services which are configured using Fail Over configuration in the APIM Publisher portal for every API’s.