We are using Terraform as IaC in our organization. I am facing an issue where we need to deploy the same set of resources for multiple groups and divisions. For example, I have ‘ABC’ as a group, which contains divisions 1, 2, 3, 4, and 5. I need to deploy to all divisions at the same time. Currently, I am using GitHub Actions with a feature called matrix, but it creates separate jobs and the billing cost is high. I want to reduce the cost of running GitHub Actions.
group=${GROUPS}
divisions=${DIVISIONS}
divisions=${divisions#[}
divisions=${divisions%]}
IFS=',' read -r -a array <<< "$divisions"
task () {
echo "Starting task for the $2 division"
cd ./infra/terraform/scheduleQueryStack
terraform workspace select -or-create "${1}_${2}_schedulequery_stack_test"
export TF_VAR_environment=${{ inputs.Stage }}
export TF_VAR_group=$1
export TF_VAR_region=${{ inputs.Region }}
export TF_VAR_division=$2
terraform apply --auto-approve
echo "Task for the $2 division completed"
}
# Array to store PIDs of background tasks
pids=()
# Max number of background tasks can be run at a time is 4.
max_concurrent=4
remove_empty_pids() {
local new_pids=()
for pid in "${pids[@]}"; do
if [ -n "$pid" ]; then
new_pids+=("$pid")
fi
done
pids=("${new_pids[@]}")
}
# Function to list all running background tasks
list_background_tasks() {
echo "PIDs of background running tasks. ${pids[@]}"
echo "Current total number of PIDs: ${#pids[@]}"
for pid in "${pids[@]}"; do
if kill -0 "$pid" 2>/dev/null; then
echo "Task with PID $pid is running"
else
echo "Task with PID $pid has completed"
# If a background task has completed, it will be removed from the array
pids=("${pids[@]/$pid}")
remove_empty_pids
echo "$pid PID removed from the PIDs array"
fi
done
}
# Run tasks in parallel using background processes and save their PIDs
for division in "${array[@]}"; do
# Check if the number of background tasks has reached the limit
while [ ${#pids[@]} -ge $max_concurrent ]; do
list_background_tasks
sleep 60
echo "Sleeping for a minute to complete the background tasks"
done
# Run the task in the background
task "$group" "$division" &
# Get the process ID of the task and add PID of background tasks to array.
pids+=($!)
done
# Function to wait for all background tasks to complete
wait_for_background_tasks() {
for pid in "${pids[@]}"; do
wait "$pid"
done
}
# Wait for all background tasks to complete
wait_for_background_tasks
echo "All tasks completed"```
Matrix is the only way to create dynamic workflows natively. Matrix value do not need to be known when a workflow is created. They can be passed into the workflow as inputs.
However, when you use a matrix, every job in a matrix runs on a separate runner in parallel. As a result, the overall workflow will complete faster but you will incur runner start-up/tear-down costs. Every runner will have to do a checkout and setup.
The only other way is to do it yourself in bash:
for division in 1 2 3 4 5; do
terraform ... $division
done
or in a custom composite action that will hide this loop.
In this case, you will run all deployments on a single runner sequentially with a single checkout and setup.
4