We are observing an interesting behavior with Orion. Our deployment is based on k3s, and we use Orion in HPA mode connected to a MongoDB in replicaset mode.
Problem: Our production cluster sends notifications (federated mode) to our secondary cluster. However, during this process, the Brokers in the secondary cluster exhibit high RAM consumption until they reach the OOMKilled state, causing the pod to restart. The issue persists even with a low number of notifications, and the RAM consumption continues to rise until the pods are restarted.
Details:
-
The problem is observed when receiving notifications via the
/op/notifyendpoint, becoming more evident with an increasing number of notifications. -
When the notification flow from the primary cluster is stopped, Orion operates normally, handling any CRUD operations without issues.
-
No anomalies or error messages are present in the logs.
-
The same behavior is observed in both Orion release 4.0 and version 3.11.
What We Have Tried:
- We are using a large number of Orion pods to minimize the number of restarts, but this has not resolved the issue.
Our Questions:
-
Is there a way to resolve this RAM consumption issue through Orion configurations?
-
Could this be an anomalous behavior of Orion, and if so, are there any known fixes or workarounds?
Any suggestions or ways to mitigate this issue would be greatly appreciated.
Thank you in advance for your help.