Often times i need to use task catchup to fill some tables, and sometimes these dags are dependent on other ones. However, dependency dag might not have runs in all days on which the dependent one will catchup.
So far solution is to merge dag with commented sensor, run catchup, then uncomment sensor.
I wonder if there is a way to specify a date from which sensor will actually check for dagruns, and until that day to have it succeeding.
Tried to make something like
@task def wait_for_newcomers_task(): if pendulum.now(tz="UTC") > pendulum.datetime(2024, 7, 31, tz="UTC"): return ExternalTaskSensor( task_id='wait_for_customer_discount_task', external_dag_id='120gp_mart_customer_discount' ) else: return True
But colleages told me it wont work.
bearmond is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.