So the root of my issue is that I have a task group with several tasks, once in a while one will fail and that will set the status of the task group to failed instantly instead of waiting for all tasks to finish.
This creates a big problem when my last task has an all_done trigger rule that gets triggered as soon as any of the tasks inside the task group fail, even though there are several tasks still running inside the task group.
How can I make Airflow wait for all tasks inside a group to finish, to set a status of the group? Ideally I’d like to make sure the task group retains a running status while there are tasks inside it running.
I’ve tried playing around with a few different configurations, but they all end up changing the task group status if one fails inside it.