Regarding Google Cloud Composer, I have defined a DAG in this way:
dataset = Dataset("//my_Dataset")
dag = DAG(
dag_id='my_dag',
default_args=default_args,
schedule=[dataset],
catchup=False)
The Dataset (//my_Dataset) can be updated by 2 different DAGs. My aim is to retrive information regarding which DAG has updated last time the Dataset. This is because my final goal is to trigger the latter DAG (my_dag) with different parameters depending on which DAG has update the Dataset.
it’s just a suggestion. I haven’t tried it myself, but you can try the following:
A triggered DAG can fetch information from the dataset that triggered it using the triggering_dataset_events template or parameter.
Reference: docs
@task
def print_triggering_dataset_events(triggering_dataset_events=None):
for dataset, dataset_list in triggering_dataset_events.items():
print(dataset, dataset_list)
print(dataset_list[0].source_dag_run.dag_id)