In our project we have different clients and identical DAGs for them with different prefixes and parameters. For instance, we have mssql_to_bigquery DAG but it’s separate for every client. That leads to the multiplication of this DAG for every client and it generates by “factory” DAG with different prefixes: Client1_mssql_to_bigquery, Client2_mssql_to_bigquery, etc.
The question is that if this “factory” method affects DAG parsing time and environment work in whole. I wonder if is this better to keep these in one DAG in multiple client tasks instead of the approach described above.
We have already tried both approaches but it’s difficult to understand what approach is better because our environment contains both of them.
Victor Tverdokhvalov is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.