I’m trying to use the official Airflow docker-compose to test a few things locally. From questions like How to pass google cloud application credentials file to docker container and Specify GOOGLE APPLICATION CREDENTIALS in Airflow I’ve tried many things and I always get an error of the type:

google.auth.exceptions.DefaultCredentialsError: Neither metadata server or valid service account credentials are found.

or (depending on what I try)

google.auth.exceptions.DefaultCredentialsError: GOOGLE_APPLICATION_CREDENTIALS path is either not found or invalid.

I am logged into GC successfully locally, and I’ve add a volume to the docker-compose with the file of the credentials, and even a conn:

version: '3.8'
x-airflow-common:
  &airflow-common
  build: .
  environment:
    &airflow-common-env
    AIRFLOW_CONN_GOOGLE_CLOUD_DEFAULT: '{"conn_type": "google_cloud_platform", "extra": {"key_path": "/opt/airflow/gc_credentials.json", "scope": "https://www.googleapis.com/auth/cloud-platform", "project": "dev", "num_retries": 2}}'

  volumes:
    - ${AIRFLOW_PROJ_DIR:-.}/dags:/opt/airflow/dags
    - ${AIRFLOW_PROJ_DIR:-.}/logs:/opt/airflow/logs
    - ${AIRFLOW_PROJ_DIR:-.}/plugins:/opt/airflow/plugins
    - ${HOME}/.config/gcloud/application_default_credentials.json:/opt/airflow/gc_credentials.json

In the Dockerfile I have:

FROM apache/airflow:2.7.3-python3.9

ENV PROJECT_ID=freshflow-dev
ENV GOOGLE_APPLICATION_CREDENTIALS=/opt/airflow/gc_credentials.json

If I enter into a container with docker exec -it scheduler-airflow-scheduler-1 bash, I indeed see the gc_credentials file and the env is correctly setup:

docker exec -it scheduler-airflow-scheduler-1 bash

airflow@45f9be08c7b7:/opt/airflow$ echo $GOOGLE_APPLICATION_CREDENTIALS
/opt/airflow/gc_credentials.json

airflow@45f9be08c7b7:/opt/airflow$ ls
airflow.cfg  dags  gc_credentials.json  logs  plugins  requirements.txt

airflow@45f9be08c7b7:/opt/airflow$ cat gc_credentials.json
{
  "client_id": "client-id.apps.googleusercontent.com",
  "client_secret": "client-secret",
  "quota_project_id": "dev",
  "refresh_token": "refresh_token",
  "type": "authorized_user"
}

How could I fix this and successfully run Operators that interact with GC? Thank you very much