I have a GRPC client which creates single channel with 4 streams(say Stream A, B, C and D). I have grpc server pods running as statefulset pods in backend. In between ingress and nginx is present for load balancing.
When 10 clients were spawned following behaviour was observed for respective configurations.
- Using default nginx configurations.
Requests were distributed at stream level, that is 4 streams created by a single client did not end up in a single server. Even the streams were distributed uniformly.
I don’t want this to happen. All streams related to a channel/client should be served by a single server pod. The distribution should happen at Client level.
-
Using nginx.ingress.kubernetes.io/upstream-hash-by:”$request_uri”
Out of 10 pods, only random 4 pods received requests. pod3 got A streams from all 10 clients. pod2 got B streams from all 10 Clients. pod6 got stream C from all 10, likewise pod7 got stream Ds. -
Using custom header: nginx.ingress.kubernetes.io/upstream-hash-by:”$req_id”
On client side added a unique serial number in the header using metadata key value pairs {“X-Request-ID” : serialID }. Distribution happened at client level. All four streams belonging to a client ended up in a single server pod. Distribution happened as below.
Pod-0 : –
Pod-1 : –
Pod-2 : client7
Pod-3 : client3, client9
Pod-4 : client5, client8
Pod-5 : client2
Pod-6 : –
Pod-7 : –
Pod-8 : client1, client10
Pod-9 : client4, client6
distribution was neither skewed nor disrtibuted equally.
How to achieve a uniform distribution like a round robin disrtibution yet maintaining channel level distribution and not distributing streams.