I have a pod part of a statefulsel with this spec:

  containers:
    - name: XYZ
...
      env:
        - name: PASSWORD
          valueFrom:
            secretKeyRef:
              name: MYSECRET
              key: password

Let’s say at start the value of the secret “MYSECRET” was “123”

Later, the value of the secret “MYSECRET” has been updated to “456”

So far it is expected the env var isn’t updated and the value within the pod is still “123”.
Now I delete the pod, on recreation, the pod environment value for $PASSWORD is still “123”.

What could cause this caching? The only solution we found so far to resolve is to drain the node then the new pod on the new node get the right value. Is there a setting to cache secret value on nodes?