I have my-app chart which is getting deployed by helm3 in the following steps. It fails saying there is another operation in progress where as the last operation was successful. When I retry the same, it is then successful. This is happening with every release that the first time it fails and retry with the same one is successful. We recently upgraded this from helm2 to helm3 via helm 2to3 convert -t tiller-admin my-dev

These are the steps:

# Step 1 - Print helm history
$ helm history --max 20 -n my-dev -o=json my-dev

This prints a json which has one release as failed but the latest one successfully deployed

[
  {
    "revision": 1,
    "updated": "2024-05-07T22:16:45.173783972Z",
    "status": "superseded",
    "chart": "my-app-0.0.81",
    "app_version": "0.0.81",
    "description": "Install complete"
  },
  {
    "revision": 2,
    "updated": "2024-05-07T23:40:22.72498084Z",
    "status": "failed",
    "chart": "my-app-0.0.81",
    "app_version": "0.0.81",
    "description": "Upgrade "my-dev" failed: cannot patch "app-job" with kind Job: Job.batch "app-job" is invalid: spec.template: Invalid value: .... field is immutable"
  },
.
.
.
  {
    "revision": 16,
    "updated": "2024-05-09T23:57:31.753789231Z",
    "status": "deployed",
    "chart": "my-app-0.0.81",
    "app_version": "0.0.81",
    "description": "Upgrade complete"
  }
]

# Step 2 - Helm upgrade
$ helm upgrade --namespace=my-dev --version=0.0.81 --values=my-dev/values.yaml --install my-dev org/my-app --debug --cleanup-on-fail --wait --timeout='7200s' --history-max 20

This is failing with the following error

upgrade.go:142: [debug] preparing upgrade for my-dev
Error: UPGRADE FAILED: another operation (install/upgrade/rollback) is in progress
helm.go:84: [debug] another operation (install/upgrade/rollback) is in progress
helm.sh/helm/v3/pkg/action.init
    helm.sh/helm/v3/pkg/action/action.go:62
.
.
runtime.goexit
    runtime/asm_amd64.s:1581

But from the helm history in step-1, the last one was successful. When I do the helm list, it shows the same

➜  ~ helm list -a -n my-dev
NAME    NAMESPACE REVISION  UPDATED                                   STATUS    CHART           APP VERSION
my-dev  my-dev    16        2024-05-09  23:57:31.753789231 +0000 UTC    deployed    my-app-0.0.81     0.0.81

So why is helm thinking there is still another operation in progress?

I have secrets as:

➜  ~ kubectl get secret -n my-dev
NAME                                          TYPE                 DATA   AGE
sh.helm.release.v1.my-dev.v1    helm.sh/release.v1   1      2d2h
sh.helm.release.v1.my-dev.v2    helm.sh/release.v1   1      2d1h
sh.helm.release.v1.my-dev.v3    helm.sh/release.v1   1      2d
.
.
sh.helm.release.v1.my-dev.v16   helm.sh/release.v1   1      51m

Where should i look to find this in progress operation. Why is the retry not giving the same error but the first try?

Thanks!