I have trouble setting up a parent pipeline in Azure Data Factory. The pipeline should only run 4 child pipelines at once however when starting the pipeline it seems to run all child pipelines at once.
Current setup:
- A pipeline which does trigger 16 other pipelines through “Execute Pipeline”-Activities
- Each child pipeline has some logic to it and does trigger a databricks notebook
- Each child pipeline can run concurrrently to the other pipelines
- Due to the activity limit and visibility each logic is in its child pipeline (cannot be combined in parent pipeline)
Wanted setup:
- Run max 4 child pipelines concurrently to avoid overloading the systems
Additionally is there a way to monitor queue times of the child pipelines? In our connected Log Analytics I was unable to find a relevant attribute.
The pipeline should only run 4 child pipelines at once however when starting the pipeline it seems to run all child pipelines at once
One of the reasons for this could be not checking the Wait on completion checkbox in the Execute pipeline activities. By checking this, the activity will wait till the completion of the current pipeline run and won’t start the next activity in the pipeline flow. So, make sure you check this in the Execute pipeline activity.
Run max 4 child pipelines concurrently to avoid overloading the systems
As you want to run all 16 pipelines from parent pipeline, you can try the below pipeline design in the parent pipeline to achieve your requirement.
You can change the dependency to after the completion of the activity instead of On success as per your requirement.
In this way, at every time, there will be only 4 child pipeline runs from the parent pipeline.
5