I’m currently using Kubernetes version 1.23.15 and I’m encountering an issue with pod prioritization using PriorityClass. I have two PriorityClass defined: low and high.
I want priority to be non preemptive.
All pods that I submit are the same, expect name and priorityClass. They all required resources to avoid Kubernetes to run it all at the same time. I also force them to be on the same node.

Here is the YAML file of a high priority pod :

apiVersion: v1
kind: Pod
metadata:
  name: high-priority-pod-1
spec:
  containers:
    - name: alpine
      image: alpine
      command: ["sh", "-c", "sleep 10"]
      resources:
        requests:
          memory: "3Gi"
          cpu: "5"
  priorityClassName: high-priority
  restartPolicy: Never
  nodeSelector:
    kubernetes.io/hostname: node1

It is the same for low-priority pod except the priorityClassName set to low-priority

And here is the YAML file for priorities

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: high-priority
  namespace: priority-test-zone
value: 1000000
globalDefault: false
preemptionPolicy: Never
---

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: low-priority
  namespace: priority-test-zone
value: 1000
preemptionPolicy: Never
globalDefault: false

Here’s the scenario: I submit a batch of 10 pods with high priority and another batch of 10 pods with low priority at the same time. As expected, the high priority pods are scheduled and run first, followed by the low priority pods.

However, I noticed an unexpected behavior when I submit some low priority pods while the high priority pods are still running. The just submitted low priority pods start running even though there are still high priority pods in the pending state.

Here is logs when you can see that other-low-priority-pod are ContainerCreating or Completed while some high-priority-pod are still in pending state.

$ k get po
NAME                        READY   STATUS              RESTARTS   AGE
high-priority-pod-1         0/1     Completed           0          66s
high-priority-pod-10        0/1     Pending             0          66s
high-priority-pod-2         0/1     Completed           0          66s
high-priority-pod-3         0/1     Completed           0          66s
high-priority-pod-4         0/1     Completed           0          66s
high-priority-pod-5         0/1     Completed           0          66s
high-priority-pod-6         1/1     Running             0          66s
high-priority-pod-7         0/1     Pending             0          66s
high-priority-pod-8         0/1     Pending             0          66s
high-priority-pod-9         0/1     Pending             0          66s
low-priority-pod-1          0/1     Pending             0          65s
low-priority-pod-10         0/1     Pending             0          65s
low-priority-pod-2          0/1     Pending             0          65s
low-priority-pod-3          0/1     Pending             0          65s
low-priority-pod-4          0/1     Pending             0          65s
low-priority-pod-5          0/1     Pending             0          65s
low-priority-pod-6          0/1     Pending             0          65s
low-priority-pod-7          0/1     Pending             0          65s
low-priority-pod-8          0/1     Pending             0          65s
low-priority-pod-9          0/1     Pending             0          65s
other-low-priority-pod-1    0/1     Completed           0          42s
other-low-priority-pod-10   0/1     Pending             0          42s
other-low-priority-pod-2    0/1     ContainerCreating   0          42s
other-low-priority-pod-3    0/1     Pending             0          42s
other-low-priority-pod-4    0/1     Pending             0          42s
other-low-priority-pod-5    0/1     Pending             0          42s
other-low-priority-pod-6    0/1     Pending             0          42s
other-low-priority-pod-7    0/1     Pending             0          42s
other-low-priority-pod-8    0/1     Pending             0          42s
other-low-priority-pod-9    0/1     Pending             0          42s

From my understanding, Kubernetes should schedule and run all high priority pods before moving on to the low priority ones. I’m not sure why a low priority pod would start running when there are high priority pods still pending.