Consider the following setup:
- A PVC, which will dynamically provision the PV
- A pod, where that PVC is mounted
I put some data on the volume mount through the pod before forcibly deleting the PV (not the PVC). The way I delete the PV is that I issue the delete command on the PV, then I edit its manifest to remove the finalizers. After that, the PVC gives me the error “Bound claim has lost its PersistentVolume. Data on the volume is lost!” However inside the pod, I can still see the files I wrote to the volume before, as well as read them. I can also write to the volume mount.
Is this something normal, or if it’s not normal, what is the mechanism behind it?
This was tested on AKS with the default storage class.
3
As explained in comment, this is a common scenario observed when you forcibly delete the PV while it’s still mounted in the pod, and mechanism behind this behavior can be explained by the delayed garbage collection of the actual physical storage resource. While Kubernetes removes the PV, the underlying storage (e.g., Azure Disk) still exists for some time and is still mounted inside the pod, allowing you to continue interacting with the data. Eventually, when the disk resource is cleaned up, the pod will lose access to the volume, resulting in data loss.
Example-
Let’s say I deployed a pod and a pvc inside my cluster
Once the pod is running, access it and write some data to the mounted volume.
I will now delete the PV forcefully to simulate the issue.
After deleting the PV, edit the PV’s manifest to remove finalizers and force Kubernetes to remove it
At this stage, the PVC should show a Lost state, but the pod should still have access to the files. If you re-verify, you will notice that the pod still has access to the volume.
Why?
This happens because the volume is still mounted, even though Kubernetes has marked it as lost.
If you want to avoid such scenarios, it’s best to avoid forcibly deleting the PV or finalizers unless you are certain about the consequences.
Reference- https://kubernetes.io/blog/2021/05/14/using-finalizers-to-control-deletion/