Airflow Scheduler Not Recognizing Unpickled DAGs Until Metadata is Manually Updated
I am working with Apache Airflow, I am creating DAG object at runtime by using some config, this code is places under dags folder where airflow schedule is executing it creating DAG object, now what I am doing is I am storing this created object in postgresql by pickling the object(because if there is no change in config there is no point in executing this entire DAG creation code), using Dill for it as airflow also uses the same, all going well but I’m experiencing an issue where the scheduler does not recognize DAGs that I retrieve from a database and unpickle, while it does recognize newly created DAG objects and also adds entry to dag_pickle table. Specifically, the behavior is as follows: