I have two pipelines:
parent pipeline to read configuration file (lookup activity), filter it with filter activity and pass the resulting array (e.g. 50 items) to ForEach activity to run child pipeline to perform per-item work in a predefined number of parallel runs (e.g. 3 at once).
The child pipeline is executing its tasks using SQL servers,
and different servers have different number of session allowed to run in parallel for this solution, e.g.
- 1 session for SQLsrv1
- 3 sessions for SQLsrv2
- 5 sessions for SQLsrv3
so I need to somehow parameterize that number of parallel runs for the child pipeline, but:
- It could be done using child pipeline’s concurrency parameter if I had only 1 SQL server,
but what to do with many if pipeline’s concurrency parameter does not accept dynamic content? - ForEach activity does not allow to parameterize its batch count either
Should I opt to:
- creating 3rd pipeline for offloading just ForEach activity to it,
- then clone this 3rd pipeline to have e.g. 5 pipelines with different number of parallel runs in ForEach activity (1…5)
- adding switch activity to parent pipeline to have e.g. 5 branches for 1…5 parallels and add ExecutePipeline activity to trigger one of those ‘3rd’ pipelines clones?
Is there a better/simpler solution?
Thanks!