Environments:
- MWAA(airflow) 2.8.1
- requirements.txt
- astronomer-cosmos==1.3.2
- apache-airflow-providers-cncf-kubernetes
- EKS(k8s) 1.27
DAG code below:
from pathlib import Path
from airflow import DAG
from pendulum import datetime
from cosmos.config import ProjectConfig
from cosmos import (
ProfileConfig,
ExecutionConfig,
ExecutionMode,
DbtTaskGroup,
)
from cosmos.profiles import RedshiftUserPasswordProfileMapping
from airflow.operators.empty import EmptyOperator
PROJECT_DIR = Path("dags/dbt/dbtcicdnew/")
with DAG(
dag_id="dbtcicdnew",
start_date=datetime(2022, 11, 27),
doc_md=__doc__,
catchup=False,
) as dag:
run_models = DbtTaskGroup(
profile_config=ProfileConfig(
profile_name="dbtcicdnew",
target_name="dev",
profile_mapping=RedshiftUserPasswordProfileMapping(
conn_id="redshift_default",
profile_args={
"schema": "public",
},
),
),
project_config=ProjectConfig(PROJECT_DIR),
execution_config=ExecutionConfig(
execution_mode=ExecutionMode.KUBERNETES,
),
operator_args={
"do_xcom_push": False,
"project_dir":"/app",
"image": "xxx.dkr.ecr.us-east-2.amazonaws.com/dbtcicdnew:1.0",
"get_logs": True,
"is_delete_operator_pod": False,
"namespace": "mwaa",
"config_file": "/usr/local/airflow/dags/kubeconfig",
"in_cluster": False,
"image_pull_policy": "Always",
},
)
e1 = EmptyOperator(task_id="pre_dbt")
e2 = EmptyOperator(task_id="post_dbt")
e1 >> run_models >> e2
trigger to run DAG.Using kubectl get po -A -w to watch task pod below:
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system aws-node-62v44 2/2 Running 0 5h48m
kube-system coredns-6dfd46db99-mszjc 1/1 Running 0 5h54m
kube-system coredns-6dfd46db99-xfrls 1/1 Running 0 5h54m
kube-system kube-proxy-c8vs2 1/1 Running 0 5h48m
mwaa dbt-task-group-base-public-category-run-4hrk5hmz 0/1 Completed 0 10s
mwaa dbt-task-group-base-public-event-run-94r4a8hr 0/1 Completed 0 9s
mwaa dbt-task-group-base-public-sales-run-6o1no285 0/1 ContainerCreating 0 7s
mwaa dbt-task-group-base-public-users-run-51toaaz5 0/1 Completed 0 7s
mwaa dbt-task-group-my-first-dbt-model-run-ouja4593 0/1 Completed 0 7s
mwaa dbt-task-group-base-public-users-run-51toaaz5 0/1 Completed 0 9s
mwaa dbt-task-group-my-first-dbt-model-run-ouja4593 0/1 Completed 0 9s
mwaa dbt-task-group-base-public-category-run-4hrk5hmz 0/1 Completed 0 12s
mwaa dbt-task-group-base-public-event-run-94r4a8hr 0/1 Completed 0 11s
mwaa dbt-task-group-base-public-users-run-51toaaz5 0/1 Completed 0 9s
mwaa dbt-task-group-base-public-category-run-4hrk5hmz 0/1 Completed 0 12s
mwaa dbt-task-group-my-first-dbt-model-run-ouja4593 0/1 Completed 0 9s
mwaa dbt-task-group-base-public-event-run-94r4a8hr 0/1 Completed 0 11s
mwaa dbt-task-group-base-public-category-run-4hrk5hmz 0/1 Terminating 0 13s
mwaa dbt-task-group-base-public-category-run-4hrk5hmz 0/1 Terminating 0 13s
mwaa dbt-task-group-base-public-event-run-94r4a8hr 0/1 Terminating 0 12s
mwaa dbt-task-group-base-public-event-run-94r4a8hr 0/1 Terminating 0 12s
mwaa dbt-task-group-base-public-users-run-51toaaz5 0/1 Terminating 0 10s
mwaa dbt-task-group-base-public-users-run-51toaaz5 0/1 Terminating 0 10s
mwaa dbt-task-group-my-first-dbt-model-run-ouja4593 0/1 Terminating 0 10s
mwaa dbt-task-group-my-first-dbt-model-run-ouja4593 0/1 Terminating 0 10s
mwaa dbt-task-group-base-public-sales-run-6o1no285 0/1 Completed 0 14s
mwaa dbt-task-group-base-public-sales-run-6o1no285 0/1 Completed 0 15s
mwaa dbt-task-group-base-public-sales-run-6o1no285 0/1 Terminating 0 16s
mwaa dbt-task-group-base-public-sales-run-6o1no285 0/1 Terminating 0 16s
mwaa dbt-task-group-base-public-listing-run-rt4ofxra 0/1 Pending 0 0s
mwaa dbt-task-group-base-public-listing-run-rt4ofxra 0/1 Pending 0 0s
mwaa dbt-task-group-base-public-listing-run-rt4ofxra 0/1 ContainerCreating 0 0s
mwaa dbt-task-group-base-public-listing-run-rt4ofxra 0/1 Completed 0 1s
mwaa dbt-task-group-my-first-dbt-model-test-lapl1fwq 0/1 Pending 0 0s
mwaa dbt-task-group-my-first-dbt-model-test-lapl1fwq 0/1 Pending 0 0s
mwaa dbt-task-group-my-first-dbt-model-test-lapl1fwq 0/1 ContainerCreating 0 0s
mwaa dbt-task-group-my-first-dbt-model-test-lapl1fwq 0/1 Completed 0 1s
mwaa dbt-task-group-base-public-listing-run-rt4ofxra 0/1 Completed 0 2s
mwaa dbt-task-group-base-public-date-run-qabzl5z5 0/1 Pending 0 0s
mwaa dbt-task-group-base-public-date-run-qabzl5z5 0/1 Pending 0 0s
mwaa dbt-task-group-base-public-date-run-qabzl5z5 0/1 ContainerCreating 0 1s
mwaa dbt-task-group-base-public-venue-run-b9ke290z 0/1 Pending 0 0s
mwaa dbt-task-group-base-public-venue-run-b9ke290z 0/1 Pending 0 0s
mwaa dbt-task-group-base-public-venue-run-b9ke290z 0/1 ContainerCreating 0 0s
mwaa dbt-task-group-base-public-listing-run-rt4ofxra 0/1 Completed 0 3s
mwaa dbt-task-group-base-public-date-run-qabzl5z5 0/1 Completed 0 1s
mwaa dbt-task-group-my-first-dbt-model-test-lapl1fwq 0/1 Completed 0 2s
mwaa dbt-task-group-base-public-listing-run-rt4ofxra 0/1 Terminating 0 4s
mwaa dbt-task-group-base-public-listing-run-rt4ofxra 0/1 Terminating 0 4s
mwaa dbt-task-group-rpt-finance-qtr-total-sales-by-event-ru-1envlgl2 0/1 Pending 0 0s
mwaa dbt-task-group-rpt-finance-qtr-total-sales-by-event-ru-1envlgl2 0/1 Pending 0 0s
mwaa dbt-task-group-rpt-finance-qtr-total-sales-by-event-ru-1envlgl2 0/1 ContainerCreating 0 0s
mwaa dbt-task-group-my-first-dbt-model-test-lapl1fwq 0/1 Terminating 0 3s
mwaa dbt-task-group-my-first-dbt-model-test-lapl1fwq 0/1 Terminating 0 3s
mwaa dbt-task-group-base-public-date-run-qabzl5z5 0/1 Completed 0 2s
mwaa dbt-task-group-base-public-date-run-qabzl5z5 0/1 Terminating 0 3s
mwaa dbt-task-group-base-public-date-run-qabzl5z5 0/1 Terminating 0 3s
mwaa dbt-task-group-my-second-dbt-model-run-7a3me2r9 0/1 Pending 0 0s
mwaa dbt-task-group-my-second-dbt-model-run-7a3me2r9 0/1 Pending 0 0s
mwaa dbt-task-group-my-second-dbt-model-run-7a3me2r9 0/1 ContainerCreating 0 0s
mwaa dbt-task-group-my-second-dbt-model-run-7a3me2r9 0/1 Completed 0 1s
mwaa dbt-task-group-my-second-dbt-model-run-7a3me2r9 0/1 Completed 0 2s
mwaa dbt-task-group-my-second-dbt-model-run-7a3me2r9 0/1 Terminating 0 3s
mwaa dbt-task-group-my-second-dbt-model-run-7a3me2r9 0/1 Terminating 0 3s
mwaa dbt-task-group-my-second-dbt-model-test-knk3wih3 0/1 Pending 0 0s
mwaa dbt-task-group-my-second-dbt-model-test-knk3wih3 0/1 Pending 0 0s
mwaa dbt-task-group-my-second-dbt-model-test-knk3wih3 0/1 ContainerCreating 0 0s
mwaa dbt-task-group-my-second-dbt-model-test-knk3wih3 0/1 Completed 0 13s
mwaa dbt-task-group-my-second-dbt-model-test-knk3wih3 0/1 Completed 0 15s
mwaa dbt-task-group-my-second-dbt-model-test-knk3wih3 0/1 Terminating 0 15s
mwaa dbt-task-group-my-second-dbt-model-test-knk3wih3 0/1 Terminating 0 15s
mwaa dbt-task-group-rpt-finance-qtr-total-sales-by-event-ru-1envlgl2 0/1 Completed 0 38s
mwaa dbt-task-group-rpt-finance-qtr-total-sales-by-event-ru-1envlgl2 0/1 Completed 0 40s
mwaa dbt-task-group-rpt-finance-qtr-total-sales-by-event-ru-1envlgl2 0/1 Completed 0 40s
mwaa dbt-task-group-base-public-venue-run-b9ke290z 0/1 Completed 0 41s
mwaa dbt-task-group-rpt-finance-qtr-total-sales-by-event-ru-1envlgl2 0/1 Terminating 0 41s
mwaa dbt-task-group-rpt-finance-qtr-total-sales-by-event-ru-1envlgl2 0/1 Terminating 0 41s
mwaa dbt-task-group-base-public-venue-run-b9ke290z 0/1 Completed 0 43s
mwaa dbt-task-group-base-public-venue-run-b9ke290z 0/1 Terminating 0 43s
mwaa dbt-task-group-base-public-venue-run-b9ke290z 0/1 Terminating 0 43s
mwaa dbt-task-group-rpt-finance-qtr-top-events-by-sales-run-cjf6gwpq 0/1 Pending 0 0s
mwaa dbt-task-group-rpt-finance-qtr-top-events-by-sales-run-cjf6gwpq 0/1 Pending 0 0s
mwaa dbt-task-group-rpt-finance-qtr-top-events-by-sales-run-cjf6gwpq 0/1 ContainerCreating 0 0s
mwaa dbt-task-group-rpt-finance-qtr-top-events-by-sales-run-cjf6gwpq 0/1 Completed 0 0s
mwaa dbt-task-group-rpt-finance-qtr-top-events-by-sales-run-cjf6gwpq 0/1 Completed 0 2s
mwaa dbt-task-group-rpt-finance-qtr-top-events-by-sales-run-cjf6gwpq 0/1 Terminating 0 2s
mwaa dbt-task-group-rpt-finance-qtr-top-events-by-sales-run-cjf6gwpq 0/1 Terminating 0 2s
Pod status changes from Pending to Completed takes 0s.
Additional infomations:
- dbt project repository
Pod Running normally.
New contributor
Lin is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.