I have a DAG running overnight where I need to run a python function on the past 3 days of data.
My schedule interval is 0 22 * * *
and my start_date is pendulum.today(tz='America/Los_Angeles').subtract(days=1)
I pass pendulum.today(tz='America/Los_Angeles).subtract(days=3)
as a parameter to my function. One issue I have is sometimes the dag runs past midnight, and when this task runs it is sometimes the next day, causing me to pass the wrong date. How can I ensure I pass a date anchored to when the dag started rather than at time of evaluation?
yoyo8855 is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.