In my DAG setup, the schedule_interval parameter has a cron expression = run 1 time per hour from 5-12 and from 13-18. And one hour (from 12-13) outside this interval.
If you manually start a task (CLEAR Task) outside of this interval, it will not work and will immediately set the SKIPPED status. And if a task started at 11:30 does not have time to complete before 12 o’clock, then it ends with the SKIPPED status.
How is it treated?

I have an Airflow DAG that runs a BashOperator task. When it fails I get an email with not much detail:

I am using docker-compose to build Airflow.

I want to dynamically create n taks, n should be defined by params, so i can define it when running the dag in the UI:

I have a PythonOperator task that requires arguments to be passed to the constructor before the python callable is invoked.

I’m rather new to airflow and couldn’t find information anywhere. Currently in my DAG i’m reading an existing Postgres connection configured in airflow thus taking the credentials for it and passing them to task that uses DatabricksSubmitNowOperator. I’m using existing cluster and i’m passing them as “base_parameters”. The process works fine but the problem is that when you open Databricks job details “base_parameters” are visible and credentials like login and password are exposed. I tried passing the credential variables as environmental variables but they are not accessible between tasks. TL DR: is there a way to read airflow connection credentials and pass them as variables to DatabricksSubmitNowOperator task without exposing them?

Have recently started to use Airflow for various jobs, trying to get handle on templating.

I have an airflow instance running inside a docker container. I might want to update Airflow in the future while ensuring that the database matches the current version of airflow.

I have an application that is essentially a web server (with a separate Dockerfile and dependencies) that includes a custom workflow engine. Additionally, I have around 20 Docker images with AI models, each with a FastAPI wrapper exposing the models’ APIs. When a user makes a request, the web server builds a DAG (not an Airflow DAG, but a DAG in this custom workflow engine), where each ‘component’ of the DAG calls the web API of a specific container with an AI model.

Is there a way to pause a task for 24 hours then run the same task again?