I am facing an issue with argoCD version v2.8.6+6f7af53 when trying to decrypt a secrets.yaml (after editing the GPG key). Following are the steps that we followed to use GPG keys to decrypt secret files:

1. Modify the argocd-repo-server deployment in argocd namespace

   # kubectl edit deployment argocd-repo-server -n argocd

   • Created these additional volumes in argocd-repo-server deployment

     volumes:
       - name: gnupg-home
         emptyDir: {}
       - name: pgp-secret-volume
         secret:
           defaultMode: 420
           secretName: pgp-secret
     

   • Added the following volumeMounts to the argocd-sidecar container in argocd-repo-server deployment

     volumeMounts:
          - mountPath: /home/pgp/
            name: pgp-secret-volume
          - mountPath: /home/argocd/.gnupg
            name: gnupg-home
     

   • Added the following env variable in sidecar container in argocd-repo-server deployment

     - name: GNUPGHOME
       value: /home/argocd/.gnupg
     

2. Created Kubernetes secret

   # gpg --export-secret-keys --armor **************************** > secrets.asc
   # kubectl create secret generic pgp-secret --from-file=./secrets.asc -n argocd
   

3. Modified the argocd-cmp-cm configmap in argocd namespace and added the following line:

   && gpg --import --batch --yes /home/pgp/secrets.asc
   

   # kubectl edit cm argocd-cmp-cm -n argocd
      discover:
        fileName: '*.yaml*'
      generate:
        command: ["/bin/sh", "-c"]
        args: ["if [ "$ARGOCD_ENV_SOP_FILE" != "" ];then sops -d sop.yaml > secrets.yaml && helm template -f values.yaml -f plainValues.yaml -f secrets.yaml -n $ARGOCD_APP_NAMESPACE $ARGOCD_APP_NAME .; elif [ "$ARGOCD_ENV_VALUE_FILE" != "" ];then helm template -f values.yaml -f plainValues.yaml -n $ARGOCD_APP_NAMESPACE $ARGOCD_APP_NAME .; else helm template -n $ARGOCD_APP_NAMESPACE $ARGOCD_APP_NAME . ;fi"]
      init:
        command: ["/bin/sh", "-c"]
        args: ["echo "$ARGOCD_ENV_VALUE_FILE" > plainValues.yaml && echo "$ARGOCD_ENV_SOP_FILE" > sop.yaml && gpg --import --batch --yes /home/pgp/secrets.asc "]
   

4. Restart the argocd-repo-server deployment

   # kubectl rollout restart deployment argocd-repo-server -n argocd
   

Initially argoCD was able to decrypt the files, but after a month GPG key got expired. Then we edited the key to extend the validity by 1 year. Then updated the kubernetes secret with the new key. After this, argoCD was not able to decrypt the files and it gave the following error.

Failed to load target state: failed to generate manifest for source 1 of 1: rpc error: code = Unknown desc = Manifest generation error (cached): plugin sidecar failed. error generating manifests in cmp: rpc error: code = Unknown desc = error generating manifests: `/bin/sh -c "if [ "$ARGOCD_ENV_SOP_FILE" != "" ];then sops -d sop.yaml > secrets.yaml && helm template -f values.yaml -f plainValues.yaml -f secrets.yaml -n $ARGOCD_APP_NAMESPACE $ARGOCD_APP_NAME .; elif [ "$ARGOCD_ENV_VALUE_FILE" != "" ];then helm template -f values.yaml -f plainValues.yaml -n $ARGOCD_APP_NAMESPACE $ARGOCD_APP_NAME .; else helm template -n $ARGOCD_APP_NAMESPACE $ARGOCD_APP_NAME . ;fi"` failed exit status 128: Failed to get the data key required to decrypt the SOPS file. Group 0: FAILED 13C31EF908C85F7E452F3223E5513A803E3341EE: FAILED - | could not decrypt data key with PGP key: | github.com/ProtonMail/go-crypto/openpgp error: Could not | load secring: open /home/argocd/.gnupg/secring.gpg: no such | file or directory; GPG binary error: exit status 2 Recovery failed because no master key was able to decrypt the file. In order for SOPS to recover the file, at least one key has to be successful, but none were.

Then I logged into the repo-server container and checked that the user did not have permission to the /home/argocd/.gnupg/ folder. As a workaround I made the following changes to the repo-server deployment under repo-server and sidecar container manifest.

securityContext:
    runAsNonRoot: false

After this the error was not observed and it was able to successfully sync the apps.

I know that this setting is not advised, how to fix the error without making changes to the securityContext?