I am trying to get kubernetes readiness probe to work for a deployment running on AKS. The cluster I am deploying to has 3 nodes with 1 tainted with node.kubernetes.io/unschedulable:NoSchedule. There are no other taints on any of the other nodes.

Let’s say the 2 nodes without the taint names are node1 and node2.

My deployment yaml look like this:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-test
spec:
  selector:
    matchLabels:
      web-app: my-test
  replicas: 2
  strategy:
    type: ~
  template:
    metadata:
      name: my-test
      labels:
        web-app: my-test
    spec:
      containers:
        - name: my-test
          image: myacr.azurecr.io/mytest:latest
          readinessProbe:
            httpGet:
              path: /api/mytest/readiness
              port: 5878
            initialDelaySeconds: 3
            periodSeconds: 3
            timeoutSeconds: 1

When I deploy the yaml with kubectl apply -f .my-deployment.yaml.

The result is the first POD goes to 1/1 Running as it should, but the second one stays as 0/1 Running. When I do command kubectl describe pod, it states Warning Unhealthy 2m26s kubelet Readiness probe failed: Get "http://10.122.0.145:5878/api/mytest/readiness": dial tcp 10.122.0.145:5878: i/o timeout (Client.Timeout exceeded while awaiting headers)

If I take out the readinessProbe from the deployment, as shown below:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-test
spec:
  selector:
    matchLabels:
      web-app: my-test
  replicas: 2
  strategy:
    type: ~
  template:
    metadata:
      name: my-test
      labels:
        web-app: my-test
    spec:
      containers:
        - name: my-test
          image: myacr.azurecr.io/mytest:latest

Both pods work with 1/1 Running, as they should. I can also tell the traffic is going to both pods. I can also tell a Pod is deployed on each node, one on node1 and the other on node2.

So now I tried adding the readinessProbe back into the deployment, this time forcing both pods on 1 node as shown below:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-test
spec:
  selector:
    matchLabels:
      web-app: my-test
  replicas: 2
  strategy:
    type: ~
  template:
    metadata:
      name: my-test
      labels:
        web-app: my-test
        spec:
      nodeSelector:
        kubernetes.io/hostname: [node1|node2]
    spec:
      containers:
        - name: my-test
          image: myacr.azurecr.io/mytest:latest

[node1|node2] This means I tested on both node1 and node2.

If I only deploy it to node1, everything works. Both Pods work and both go to state 1/1 Running

If I only deploy it to node2, neither work. When doing command kubectl describe pod on both pods, it states 0/1 Running with error Warning Unhealthy 2m26s kubelet Readiness probe failed: Get "http://10.122.0.145:5878/api/mytest/readiness": dial tcp 10.122.0.145:5878: i/o timeout (Client.Timeout exceeded while awaiting headers)

I am really not sure why node2 is not able to respond to the readinessProbe like node1. If I deploy without the readinessProbe, node2 responds to http request without an issue.

What exactly am I doing wrong here?