Hello fellow DevOps Architects!

What exactly am I even asking?

I’ve come up with a proposed approach which includes 42 Build Pipelines and a further 6 release pipelines. Maybe this is just how “big software” works – and it’s totally fine and exactly how this is supposed to work? ????‍♂️

But equally – maybe I’m missing a trick, and there is a much better way of doing this? I’d basically like a bit of a sanity check on this.

Intro

I am considering a re-design of our build and release pipelines for a fairly large and complex solution – all of which is hosted on Microsoft Azure. This consists of:

• A single Shared App Service
• 3x Web Apps
• 2x Function Apps
• Multiple overlapping config components (AppConfig / KeyVault, EventGrid, Redis Cache, AI Search, Storage Accounts and CosmosDBs .. etc etc)
• An “Infrastructure” project which provisions all of the cloud services
  (we are using Pulumi, but for this example could just as easily be
  ARM or Bicep based).

The problem space

One of the issues we have faced is major features overlapping during development.

We need to be able to push these into separate environments for testing BEFORE we merge them into our main RELEASE branch – otherwise we end up with a big monolithic code base which includes both “ready to release” as well as “buggy / awaiting testing” code at the same time.

To complicate this – the solution contains multiple overlapping disciplines; React, C#, Python, Machine Learning .. and with a lot of moving parts (e.g. Cognitive Services and Azure Search services) so it’s unlikely that people can dev and test locally on just their own box. We need to bring stuff together in order to be able to integration test stuff.

We basically need to have multiple full end-to-end “build > deploy > test” lifecycles available so that code merges don’t end up getting muddled – and allowing our acceptance testing to independently test features for release.

We ALSO want to be able to release each component with a fair degree of independence – so we can push out a single web app without releasing code to everything else.

Proposed Architecture

So I’ve suggested an initial approach with 6 different code branches for EACH Repo that we use:

each of these would have a corresponding hosted cloud environment we can deploy to

• RELEASE (for the main production release process into formal TEST / STAGE / LIVE)
• PATCH (for patching emergent bugs on live code)
• RC1 (Release Candidate Branch)
• RC2 (Release Candidate Branch)
• RC3 (Release Candidate Branch)
• DEV1 (Dev Branch)
• DEV2 (Dev Branch)
• DEV3 (Dev Branch)

The idea for the RC and DEV branches is that it allows us to collaboratively develop multiple major features simultaneously, but independently (in the DEV1, DEV2, DEV3 branches) – and these can then be released to testers (in the RC1, RC2, and RC3 environments).

Why do we need specifically named branches?

.. why don’t we just create feature branches on the fly?

Well – this is driven partly from both Auth0 and Pulumi. Both of which require specific named environments (on the Auth0 side we need to define the callback URLs for 0Auth to work correctly), so we need these to be defined in advance.

It also makes testing a lot easier because we have Custom Domains setup for everything.

So what exactly feels wrong about this?

For me .. it’s the scale.

We currently have six different build artifacts (3x Web Apps, 2x Function Apps, 1x Infrastructure). This is likely to grow going forward.

We also have 8 branches, and again this might grow going forward.

So that is 42 separate build pipelines (6x artifacts, in 8x branches) – and a further 6x release pipelines (one for each branch – triggered by the artifact drops).

Is that .. too many? Is that how many you are supposed to have? Are there some people out there going “LMAO .. we have hundreds! this is small fry!” or am I missing something?

Should I be looking at a more clever way of branching, building, and merging the code?

…

I guess what I’m really looking for is a nod and sanity check that I’m not doing anything crazy!

If you’ve managed to reach the end of this post then I very much thank you for your time – and would appreciate your thoughts!