I’d like to have an Airflow DAG get some status information back after running an AzureContainerInstancesOperator task in Airflow. With the equivalent docker operator, DockerOperator, it is possible to get the last line of output by default inserted into XCOM. With the AzureContainerInstancesOperator, this seems less likely to be possible since the container is run remotely, possibly without network access.

I have a dag scheduled using EventsTimetable. It ran three time on schedule as expected, and then completely ignored two runs in a row- No failure/skipped in the UI, no alert email. Nothing in the logs either, as if the run was never scheduled at all. The “next run” showed up correctly in the UI before the runs that were skipped. Today, it magically started running on schedule again. We are running airflow on AKS, but I checked the pods and all seemed to be running normally when the outage occurred.

We have a Airflow job thats sends a lot of messages (>100k) to SQS. The job fails sometimes fails for no reason. When retriggered, it succeeds without any change in code.
Error message: airflow.exceptions.AirflowException: Bash command failed. The command returned a non-zero exit code 1.

I am starting to learn how to use Airflow for pipelines.
I have the simplest DAG in

I’m trying to build my first airflow DAG in k8s running on custom image.
I did the following example code:

Often times i need to use task catchup to fill some tables, and sometimes these dags are dependent on other ones. However, dependency dag might not have runs in all days on which the dependent one will catchup.
So far solution is to merge dag with commented sensor, run catchup, then uncomment sensor.
I wonder if there is a way to specify a date from which sensor will actually check for dagruns, and until that day to have it succeeding.

I use airflow to schedule my ETL jobs, and the project have version control.

I have an airflow DAG that should run on 1 AM every day. Now from the task details it does seems like it ran on 1 AM on the 24th (the time of this post):

Is there an accepted best practice for passing or accessing encrypted Airflow variables/connections in modules executed during the DagRun? Is passing it as an argument to the PythonOperator with op_kwargs sufficient?

So I’ve updated the value of job_heartbeat_sec in the [scheduler] section of airflow.cfg. I stopped and restarted the scheduler after updating the value of the variable. I expected the /health endpoint of the Webserver UI to only update the heartbeat timestamp every 20 seconds, but it is still using the default 5.