I have kubernetes CRON job that invokes simple script that is cloning git repository to persistent storage. Job is invoked every 15 minutes and once per day or two I have job that is not finishing and it has status ‘Terminating’.
│ Warning FailedKillPod 14h (x94 over 2d10h) kubelet error killing pod: [failed to "KillContainer" for "pull-repo" with KillContainerError: "rpc error: code = DeadlineExceeded desc = an error occurs during waiting for container "3ff92cf6a48e46345e635a5bbc8f20d334c84 │
│ 927b7e700076f9c795799029d19" to be killed: wait container "3ff92cf6a48e46345e635a5bbc8f20d334c84927b7e700076f9c795799029d19": context deadline exceeded", failed to "KillPodSandbox" for "34ceec99-8a5e-45b8-a298-cdf2ae8e43df" with KillPodSandboxError: "rpc error: code = Dead │
│ lineExceeded desc = context deadline exceeded"] │
│ Warning FailedKillPod 7h20m (x110 over 2d10h) kubelet error killing pod: [failed to "KillContainer" for "pull-repo" with KillContainerError: "rpc error: code = DeadlineExceeded desc = context deadline exceeded", failed to "KillPodSandbox" for "34ceec99-8a5e-45b8-a298- │
│ cdf2ae8e43df" with KillPodSandboxError: "rpc error: code = DeadlineExceeded desc = failed to stop container "3ff92cf6a48e46345e635a5bbc8f20d334c84927b7e700076f9c795799029d19": an error occurs during waiting for container "3ff92cf6a48e46345e635a5bbc8f20d334c84927b7e700076f9 │
│ c795799029d19" to be killed: wait container "3ff92cf6a48e46345e635a5bbc8f20d334c84927b7e700076f9c795799029d19": context deadline exceeded"] │
│ Warning FailedKillPod 5h49m (x446 over 2d10h) kubelet error killing pod: [failed to "KillContainer" for "pull-repo" with KillContainerError: "rpc error: code = DeadlineExceeded desc = context deadline exceeded", failed to "KillPodSandbox" for "34ceec99-8a5e-45b8-a298- │
│ cdf2ae8e43df" with KillPodSandboxError: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"] │
│ Warning Unhealthy 4m52s (x21033 over 2d10h) kubelet (combined from similar events): Readiness probe errored: rpc error: code = Unknown desc = failed to exec in container: failed to start exec "1498eacd7dc6dfb402d060fb64114336fb9994060c9ac718aea16c3bd92f6fd8": OCI ru │
│ ntime exec failed: exec failed: cannot exec in a stopped container: unknown
When I try to delete this pod my command also hangs and only thins that is working is delete with force.
Any ideas what can be source of this behavior?