We have a several pipelines that deploy to 2 AKS clusters which has been working for about 6 months. Suddenly, last Thursday, all the deployments started to fail. The pipeline is set up to delete the last deployment before it deploys the updated code. The AKS cluster is private and the deployment happens from a self-hosted agent machine. Here is what the step looks like

        strategy:
          runOnce:
            deploy:
              steps:
                - task: KubernetesManifest@1
                  displayName: Delete previous deployment
                  inputs:
                    connectionType: "azureResourceManager"
                    azureSubscriptionConnection: "$(aksServiceConnection)"
                    azureResourceGroup: $(askResourceGroupName)
                    kubernetesCluster: $(aksName)
                    namespace: $(namespaceDv)
                    action: delete
                    arguments: deployment mynamespace-deploy --ignore-not-found=true   

However the pipeline fails with the following error:

##[debug]MSAL - ServicePrincipal - clientSecret is used.
##[debug][Thu, 27 Jun 2024 15:03:52 GMT] : [] : @azure/[email protected] : Info - acquireTokenByClientCredential called
##[debug][Thu, 27 Jun 2024 15:03:52 GMT] : [9a5f7d9c-befc-48a8-aede-4f35e9f59201] : @azure/[email protected] : Info - Building oauth client configuration with the following authority: https://login.microsoftonline.com/xxxxxxxxxxxxxxx/oauth2/v2.0/token.
##[debug][Thu, 27 Jun 2024 15:03:52 GMT] : [9a5f7d9c-befc-48a8-aede-4f35e9f59201] : @azure/[email protected] : Info - Sending token request to endpoint: https://login.microsoftonline.com/xxxxxxxxxxxxxxx/oauth2/v2.0/token
##[debug]MSAL - retrying getMSALToken - temporary error code: network_error
##[debug]MSAL - retrying getMSALToken - remaining attempts: 3
##[debug]Agent environment resources - Disk: / Available 53442.00 MB out of 68979.00 MB, Memory: Used 521.00 MB out of 11957.00 MB, CPU: Usage 6.34%
##[debug]MSAL - getMSALToken called. force=undefined
##[debug][Thu, 27 Jun 2024 15:03:54 GMT] : [] : @azure/[email protected] : Info - acquireTokenByClientCredential called
##[debug][Thu, 27 Jun 2024 15:03:54 GMT] : [a2b748f6-2067-4bd1-841c-d1c386f57725] : @azure/[email protected] : Info - Building oauth client configuration with the following authority: https://login.microsoftonline.com/xxxxxxxxxxxxxxx/oauth2/v2.0/token.
##[debug][Thu, 27 Jun 2024 15:03:54 GMT] : [a2b748f6-2067-4bd1-841c-d1c386f57725] : @azure/[email protected] : Info - Sending token request to endpoint: https://login.microsoftonline.com/xxxxxxxxxxxxxxx/oauth2/v2.0/token
##[debug]MSAL - retrying getMSALToken - temporary error code: network_error
##[debug]MSAL - retrying getMSALToken - remaining attempts: 2
##[debug]MSAL - getMSALToken called. force=undefined
##[debug][Thu, 27 Jun 2024 15:03:56 GMT] : [] : @azure/[email protected] : Info - acquireTokenByClientCredential called
##[debug][Thu, 27 Jun 2024 15:03:56 GMT] : [22e38295-49ce-40c4-b4a5-6d59afb0e24a] : @azure/[email protected] : Info - Building oauth client configuration with the following authority: https://login.microsoftonline.com/xxxxxxxxxxxxxxx/oauth2/v2.0/token.
##[debug][Thu, 27 Jun 2024 15:03:56 GMT] : [22e38295-49ce-40c4-b4a5-6d59afb0e24a] : @azure/[email protected] : Info - Sending token request to endpoint: https://login.microsoftonline.com/xxxxxxxxxxxxxxx/oauth2/v2.0/token
##[debug]MSAL - retrying getMSALToken - temporary error code: network_error
##[debug]MSAL - retrying getMSALToken - remaining attempts: 1
##[debug]Agent environment resources - Disk: / Available 53442.00 MB out of 68979.00 MB, Memory: Used 521.00 MB out of 11957.00 MB, CPU: Usage 6.16%
##[debug]MSAL - getMSALToken called. force=undefined
##[debug][Thu, 27 Jun 2024 15:03:58 GMT] : [] : @azure/[email protected] : Info - acquireTokenByClientCredential called
##[debug][Thu, 27 Jun 2024 15:03:58 GMT] : [cf0db17c-ccb6-41f1-a3a5-a67c2698a467] : @azure/[email protected] : Info - Building oauth client configuration with the following authority: https://login.microsoftonline.com/xxxxxxxxxxxxxxx/oauth2/v2.0/token.
##[debug][Thu, 27 Jun 2024 15:03:58 GMT] : [cf0db17c-ccb6-41f1-a3a5-a67c2698a467] : @azure/[email protected] : Info - Sending token request to endpoint: https://login.microsoftonline.com/xxxxxxxxxxxxxxx/oauth2/v2.0/token
##[warning]Can't find loc string for key: CantDownloadAccessProfile
##[debug]Processed: ##vso[task.issue type=warning;source=TaskInternal;]Can't find loc string for key: CantDownloadAccessProfile
##[debug]task result: Failed
##[error]CantDownloadAccessProfile mycluster-aks Could not fetch access token for Azure. Status code: network_error, status message: Network request failed  

Steps done:

• Ensured that client secret is not expired

• Logging into the Agent machine, I tested a curl to login.microsoftonline.com using the client secret and it returns a token

• Logging into the Agent machine, I tested kubelogin using the client secret and it returns a token

• updated kubectl and kubelogin to the latest version

However, the failure remains.

Any ideas?

Thanks in advance!