I am running structured spark streaming job and using EmrContainerOperator to trigger the spark job.
My requirement is to re-trigger airflow task which has been created using EmrContainerOperator every time the task fails. This needs to be achieved programmatically.
Intention is to create infinite loop and re-trigger the task every time it fails. How this can be accomplished? I am using Airflow verion 2.7.2
I see options to re-trigger DAG, but my requirement is to re-trigger tasks every time it fails.