I am facing a significant issue with my self-hosted build agents in my CI/CD pipeline, and I need a comprehensive solution. Here are the details:

Build Agents and Services:

I have 6 build agents.
I manage 30-40 services in the pipeline.

Problem:

During the build process, my self-hosted build agents’ disks are becoming full due to the accumulation of Docker images and cache layers.
We are already using the scratch base image to minimize the size of the Docker images.

Requirements:

I need a solution that can preserve the cache layers to speed up the build process but also ensures that the disk does not get full.

What I Tried and Expected Results

I have attempted several solutions to address the issue, but none have been entirely satisfactory:

Docker System Prune -a 

Command: docker system prune -a
Expected Outcome: Free up disk space by removing all unused data (containers, networks, images, and cache).
Actual Result: This solution significantly slowed down the build process as it removed essential cache layers.

Docker Image Removal in Azure Pipeline:

Command:
yaml

bash: |
  docker rmi img/address/image_name:$(Build.BuildId)
displayName: "Docker Delete"

Expected Outcome: Remove specific images to free up space.
Actual Result: This command did not free up enough space to prevent the disk from becoming full.
Desired Solution
I need a solution that keeps the build process fast and prevents the disk from getting full. It is acceptable to delete images and containers if necessary.

Please provide a solution that addresses the following:

Maintaining Build Speed: Ensure that the build process remains efficient and quick.

Effective Disk Space Management: Implement a strategy to manage disk space effectively.

Preserving Cache Layers: Find a way to keep essential cache layers while freeing up disk space.

Automation and Maintenance: Utilize scripts or tools to automate the cleanup process without manual intervention.

Can you help with a solution that meets these requirements?