I have an authenticated workload running on my GKE cluster that is using workload identity.

I can confirm that workload identity engages correctly using the service account, because the secrets-store CSI provider pulls in the secrets given the correct permissions.
(Removing “Secret Accessor” role in turn results in an error as expected)

The workload itself consists of a main node.js container that is trying to authenticate via ADC to BigQuery:

import { BigQuery } from '@google-cloud/bigquery';

const projectId = 'pixels-k8s';
const bq = new BigQuery({
  projectId,
});

and an ESP container sidecar that is set up according to this documentation from GCP

IAM Roles that are associated with the workload identity account are as follows:

BigQuery Admin
Cloud Endpoints Service Agent
Cloud Trace Agent
Secret Manager Secret Accessor
Service Controller

so I think the service account should have all permissions neccessary to work.

Authenticating to BigQuery results in a 403 with the message "ACCESS_TOKEN_SCOPE_INSUFFICIENT"

Authenticating to ESP results in following log message: fail to initialize config manager: http call to GET https://servicemanagement.googleapis.com/v1/services/data-piper-pixels-backend.endpoints.pixels-k8s.cloud.goog/rollouts?filter=status=SUCCESS returns not 200 OK: 403 Forbidden

I’ve tried to give excessive permissions to the WI for both ESP and BigQuery to no avail.
I’ve also associated a GSA with the workload identity (the association worked, confirmed with the secrets store) and tried it, again to no avail.

What am I doing wrong?