We have an Azure DevOps pipeline that executes functional and UI tests.
To achieve that, as a first step the pipeline deploys our application (which consists of a SQL database and multiple ASP.NET web sites) to a dedicated environment, then executes the tests against that.
This means there can be only one executing test pipeline run at any given time, as it needs that environment and multiple parallel runs would overwrite each other’s application deployments or would interfere with each other in other ways.
As this test pipeline takes some time to execute we wanted to parallelize its runs somehow, and for now we went with the naïve approach: we made a new “test environment 2”, then cloned the the pipeline as “test pipeline 2”, and there we had the possibility to execute two at the same time.
This approach works, but has drawbacks: you need to “manually load balance” and chose the pipeline that is free or has a shorter queue; the pipeline yaml is duplicated code; even though the two pipelines do the same thing they’re shown as different distinct pipelines, etc.
We know and accept the fact that we’ll need as many test environment as many parallel runs we want, but other than that, is there a way to improve this solution?
Ideally we should have only one pipeline, that automatically selects from the available environments.
8
I think I see where you’re going with this.
For YAML-based pipelines, instead of cloning the pipeline and changing values, we would normally use a multi-stage pipeline where each “stage” represents a different environment. For each stage environment, we’d create a corresponding Azure Pipeline Environment (Pipelines > Environments) and the assign an Approval check to pause and require approval before executing that stage. Using a lockBehavior on the stage ensures that only one stage of this kind is active at a time.
This would allow you to have a single pipeline with multiple test environments, and you’d approve which environment to use:
stages:
- stage: build
- stage: test1
dependsOn: build
lockBehavior: sequential
jobs:
- deployment: test
environment: test1
strategy:
runOnce:
deploy:
steps:
- script: echo 'execute tests'
- stage: test2
dependsOn: build
lockBehavior: sequential
jobs:
- deployment: test
environment: test2
strategy:
runOnce:
deploy:
steps:
- script: echo 'execute tests'
Another alternative would be to have a single stage, but pick the environment at queue-time. You would also set an Exclusive Lock check on the Azure Pipeline Environment to ensure the environment has only one active pipeline:
parameters:
- name: testEnvironment
type: string
values:
- test1
- test2
default: test1
stages:
- ...
- stage: test
dependsOn: build
jobs:
- deployment: test
environment: ${{ parameters.testEnvironment }}
...
However, the management of which environment to execute against becomes a human effort when approving the environment or when queuing — which is what you’re trying to avoid. It sounds like you’re looking for a mechanism to load-balance executions across different environments so that the “test” stage just picks an environment that is available. Removing human-effort is an automation problem and there are probably many different ways to achieve this.
Perhaps the most direct way is to use the DeploymentRecords endpoint in the Azure DevOps REST API to determine which environment is available and then set a variable that defines which environment to use. However, because the environment attribute of the deployment job must be defined at compile time, you would define multiple jobs and use a condition for the appropriate environment:
- stage: test
dependsOn: build
jobs:
- job: checkEnv
steps:
- task: PowerShell@2:
name: QueryEnv # needed for job dependencies
displayName: 'Determine Test Environment'
script: |
# use REST API to query which environment is actively running/available
$envName = ...
# set a variable
Write-Host "##vso[task.setvariable variable=TestEnvironment;isOutput=true]$envName"
# conditionally use this environment based on 'checkEnv' job output
- deployment: test1
dependsOn: checkEnv
environment: test1
condition: $[ eq(dependencies.checkEnv.output['QueryEnv.TestEnvironment'], 'test1') ]
strategy:
runOnce:
deploy:
- steps: ...
# conditionally use this environment based on 'checkEnv' job output
- deployment: test2
dependsOn: checkEnv
condition: $[ eq(dependencies.checkEnv.output['QueryEnv.TestEnvironment'], 'test2') ]
environment: test2
strategy:
runOnce:
deploy:
- steps: ...
There will be edge conditions, like what happens if both environments are actively in use. You could pick the environment with the earliest pipeline, but this won’t guarantee that this will be the next available environment. If you wanted to guarantee the next available environment, you could move the decision into a Function App and use the Function App check to periodically query your environments and wait until it becomes available before your pipeline executes.
1