I am running a self-managed Microk8s Kubernetes with Calico CNI. I have recently added a new node into the cluster and I have been seeing some network problems between pods on particular nodes.
I have been able to diagnose that the problem occurs when only TCP is allowed in the firewall rules. For some reason, when UDP traffic is not allowed through the firewall, intra-cluster TCP traffic stops working (I get timeouts from curl).
I am testing three different scenarios:
- Direct network traffic (one server to nginx on another server, no Kubernetes involved;
Host -> Host) - Request sent from one server to a NodePort service exposing a pod in on another server (
Host -> Kubernetes) - Request sent from a pod on one Node to a pod on another node (
Kubernetes -> Kubernetes)
These are the results when only one of TCP or UDP protocols are allowed:
- Firewall allows only TCP traffic
- Host -> Host ✅
- Host -> Kubernetes ✅
- Kubernetes -> Kubernetes ❌
- Firewall allows only UDP traffic
- Host -> Host ❌
- Host -> Kubernetes ❌
- Kubernetes -> Kubernetes ✅
What is going on here? Somehow it seems that when intra-Kubernetes communication happens, the whole TCP communication is tunneled through UDP.
I searched for Kubernetes or Calico needing the UDP ports to communicate but nothing eye-catching came up. Is this working as expected and do I need to always open both TCP and UDP?
Thank you