I have set up a minimal version of a Composer environment in GCP. Through Terraform, I have deployed this thus far:
resource "google_composer_environment" "airflow-poc-env" {
name = "airflow-poc-environment"
region = var.region
config {
software_config {
image_version = "composer-2.8.4-airflow-2.7.3"
env_variables = {
DWH_PROJECT="VALUE"
}
}
node_config {
service_account = module.cloud-composer-service-account.service_account.email
}
}
}
Now, I want to give the Airflow environment certain variables that I can use when defining a task within a DAG, e.g.
start_cloud_run_job = CloudRunExecuteJobOperator(
task_id='<task-id>',
region='<region>',
project_id=Variable.get("DWH_PROJECT"),
job_name='<job-name>',
)
As provided in the Terraform documentation, I suppose that the env_variables
block should provide key-value pairs of variables in the Admin > Variables section in the Airflow UI. Or is this not the case and should you always provide those variables manually? Below the section I mean.
Airflow Variables section
I tried to provide key-value pairs within in the Terraform config, expecting that this will deploy environment variables in the section above.
Or maybe another question: can those environment variables be set via Terraform or should you provide them manually or via the Variable.set(key="DWH_PROJECT", value="VALUE", serialize_json=<bool>)
?
Mattheo is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.