I am trying to deploy a speech to text (STT) model on Kubernetes, but I am having some limitations due to the model nature. The STT model only accepts one session/connection. What I mean here is that no more than one client can connect to the model at the same time, or the model will just fail to process the new connection and eventually crash.

The model saves past inputs and outputs (or you can call it model states) and combines it with new inputs to generate the new outputs. Thus, the model was limited to accept one connection so not to combine and corrupt the saved states. I can’t edit the model as it is a compiled file.

I headed to Kubernetes so that I can expand the STT services per clients and shrink it to save resources when no one is requesting.

What I am looking for is that each new client/IP connects the STT service should be routed to a single pod containing the STT model that only serves this specific client.

I have tried the built-in HPA in Kubernetes, but it is limited to resource usage only (CPU & Memory) which doesn’t fix the issue in my case. The first client would be routed to a pod and starts a session and just works fine. By connecting the second client, the first request would just be sent to the same pod of the first client (resource limits weren’t hit yet) and this crashes the model, this leads to pod restart. Then both clients will be routed to another pod (most probably the same pod again ????) and so on …

I have searched for a solution for this problem and actually found some:

• Use Kubernetes Event driven Autoscaling (KEDA)
• Use KubeFlow
• Migrate the model to a framework that will handle everything

Understanding, studying, and trying each solution is very hard especially that I don’t have experience in any of them and none is guaranteed to fix my problem.

Do you have any ideas or solutions?