I have a cluster of 3 machines IPs ending with .6, .8, .9 and have a replica of the machine running on these pods. The call to the pod is only successful when it happens on .6, initially i thought something is wrong on the 8 and 9 node. I deleted them and recreated but still same issue. Then i tried deleting all 3 nodes and create 3 new, I noticed the same only one node was working and others were failing.

When I get endpoints
kubectl -o yaml endpoints
I noticed that the endpoints for the deployment, has proper ip and nodeName(nodeName is shown as the ip of the node). The strange thing I noticed is the call is only successful if the node is the first one in the list of addresses in the endpoint all the rest timeout. When I go to individual pods which give timeout, I see they are working and the pod is healthy and a local curl call works as expected. Any idea what I may be doing wrong or what can be done to debug further?