I am setting up a CI/CD pipeline through Azure DevOps to test my Terraform code, my organization is planning on switching to CI/CD for our infrastructure.
I am trying to validate Terraform in my Azure tenant through a Service Connection in a YAML Pipeline:

name: Azure Infrastructure CI/CD

trigger:
  branches:
    include:
      - main

pool:
  vmImage: ubuntu-latest

variables:
  - name: public_key
    value: $(public_key)
  - name: terraformSecret
    value: $(client_secret)

steps:
  - checkout: self
    submodules: true

  - task: TerraformInstaller@0
    inputs:
      terraformVersion: 'latest'

  - script: az login --service-principal -u "$(client_id)" -p $(client_secret) --tenant "$(tenant_id)"
    displayName: 'Azure CLI Login'

  - script: az account set --subscription "$(subscription_id)"
    displayName: 'Azure Subscription Set'
  
  - script: |
      terraform init
      terraform plan -out=tfplan 
        -var="public_key=${public_key}" 
        -var="client_secret=${terraformSecret}"
    displayName: 'Terraform Init and Plan'
    workingDirectory: .

  - script: |
      terraform apply -auto-approve tfplan
    displayName: 'Terraform Apply'
    workingDirectory: .

Here is my Terraform provider file:

provider "azurerm" {
    features {}

    client_id       = "xxxxx"
    client_secret   = var.client_secret
    tenant_id       = "xxxxx"
    subscription_id = "xxxxx"
}

And an excerpt from my variables Terraform variables file:

variable "client_secret" {
  type        = string
}

And I have all environment variables set correctly on the pipeline in Azure DevOps.
I am getting this error:

Planning failed. Terraform encountered an error while generating this plan.

╷
│ Error: building AzureRM Client: Authenticating using the Azure CLI is only supported as a User (not a Service Principal).
│ 
│ To authenticate to Azure using a Service Principal, you can use the separate 'Authenticate using a Service Principal'
│ auth method - instructions for which can be found here: https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/guides/service_principal_client_secret
│ 
│ Alternatively you can authenticate using the Azure CLI by using a User Account.
│ 
│   with provider["registry.terraform.io/hashicorp/azurerm"],
│   on providers.tf line 10, in provider "azurerm":
│   10: provider "azurerm" {
│ 
╵

##[error]Bash exited with code '1'.

I have checked the Terraform docs, and they don’t show how I can authenticate a terraform connection through a script, and I want to protect the client secrete as much as I can, and not make it available for everyone accessing this code. I have also used a bunch of LLMs and the insight is minimally helpful as with most terraform-related projects.

How do I authenticate my terraform code through a service principal on a YAML pipeline?