I am implementing a tumbling window trigger for an Azure Data Factory (ADF) pipeline that triggers two Databricks jobs and one Power BI refresh activity. These three activities run sequentially. My ADF pipeline should only run if the previous run has succeeded. The refresh should happen about every 30min.
Here is the configuration for my trigger:
{
"name": "TRIGGER",
"properties": {
"annotations": [],
"runtimeState": "Started",
"pipeline": {
"pipelineReference": {
"referenceName": "TestTumblingWindowTriggerPipeline",
"type": "PipelineReference"
}
},
"type": "TumblingWindowTrigger",
"typeProperties": {
"frequency": "Minute",
"interval": 30,
"startTime": "2024-12-10T15:51:00Z",
"delay": "00:00:00",
"maxConcurrency": 1,
"retryPolicy": {
"intervalInSeconds": 30
},
"dependsOn": [
{
"type": "SelfDependencyTumblingWindowTriggerReference",
"offset": "-0.00:30:00"
}
]
}
}
}
- What happens if my ADF jobs start to exceed the interval and offset? Will they queue up?
- What do I need to do to “clear” a queue? Do I need to redeploy the pipeline?
- Do you have experience with such a scenario? If so, is there something else you would recommend monitoring?
Any insights or recommendations would be greatly appreciated. Feel free to point me to relevant documentation, I may have missed something.
2