I have 2 kubernetes clusters deployed almost the same way with kubespray.

On cluster a, the control node is on the same cloud as the worker nodes. On this cluster I can curl services by cluster-ip:endpoint from any node, including the control node.

On cluster b, the control node is on a different cloud. This cluster works fine except the control node can’t curl endpoints for pods running on the workers on their cluster-ip:endpoint. I can curl those fine from any of the workers however. I also can’t curl endpoints for pods running on the control node from workers, but can curl them from the control node.

The kube-ipvs0 interface looks the same on control node and workers:

3: kube-ipvs0: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN group default 
link/ether 02:f5:5c:01:a2:ea brd ff:ff:ff:ff:ff:ff
inet 10.233.0.1/32 scope global kube-ipvs0
   valid_lft forever preferred_lft forever
inet 10.233.0.3/32 scope global kube-ipvs0
   valid_lft forever preferred_lft forever
inet 10.233.4.96/32 scope global kube-ipvs0
   valid_lft forever preferred_lft forever
inet 10.233.47.117/32 scope global kube-ipvs0
   valid_lft forever preferred_lft forever
inet 10.233.21.11/32 scope global kube-ipvs0
   valid_lft forever preferred_lft forever
inet 10.233.52.127/32 scope global kube-ipvs0
   valid_lft forever preferred_lft forever
inet 10.233.13.95/32 scope global kube-ipvs0
   valid_lft forever preferred_lft forever
inet 10.233.0.10/32 scope global kube-ipvs0
   valid_lft forever preferred_lft forever
inet 10.233.29.235/32 scope global kube-ipvs0
   valid_lft forever preferred_lft forever
inet 10.233.32.105/32 scope global kube-ipvs0
   valid_lft forever preferred_lft forever

I’m using calico which has a pod running on each node, as does kube proxy. I tried opening all traffic on the worker nodes firewall temporarily, that didn’t change anything.

Does someone have an explanation?

Thanks for reading!