I have a Spring Cloud Gateway application (reactive Netty) that runs in several GCP GCE VMs. It has an average 500 tps traffic per VM. After several weeks of running smoothly the gc young memory allocated spikes 10x from one minute to the other, randomly, in all VMs in a time span of some hours. Some time after the memory allocation spike (minutes to hours) there’s a severe performance degradation, with the server response times (spring.cloud.gateway.requests) and backend call times (http.client.requests) increasing up to 20x the normal values. CPU usage increases moderately from about 10% up to 25%. Memory usage also increases moderately from 43% up to 54%. An application restart resolves the issue. Same issue has happened twice in 7 months.

Environment:

• Java Version: 21
• Spring Cloud Gateway Version: 4.1.5
• Netty Version: 1.1.21
• JVM GC settings: Same issue happened with G1 and Parallel
• Traffic/Payload: Stable traffic and payload size.

Observed abnormal metrics:

• jvm/gc/memory/allocated: Spike by 10x with the same traffic and
  payload. It’s the first symptom of the issue.
• reactor/netty/bytebuf/allocator/active/direct/memory: From 100k to
  7M.
• reactor/netty/eventloop/pending/tasks: Spiked from 0/1 to over
  300.
• reactor/netty/http/client/address/resolver: Spiked from 2ms to 500ms.
• reactor/netty/http/client/tls/handshake/time: Went from 5ms to
  1500ms.
• jvm/threads/states: Blocked threads goes from 0 to 80.
• TCP Connections: Active connections spiked from 20 to 300.

I’ve been trying to reproduce this issue since it first happened to no avail. I also reviewed every metric, tried searching anything related on the internet, read everything I could find about Garbace collector and memory leaks in Spring Boot / Spring Cloud Gateway / Netty, asked chatgpt and bard, but I couldn’t be able to solve this mistery.

Any insights or suggestions are highly appreciated!