Disclaimer : I know almost nothing about DevOps and have been tasked with implementing CI/CD for this project. My question may be unclear and lacking in detail, so any guidance is appreciated. Please, gently correct me and tell me what I can do to improve my question 🙂

I am using PHPUnit 10.5 to test my project in php 8.1, which is hosted on GitLab. The project is deployed on a bunch of different servers, including a test server, and many production servers.

Question

While configuring the YAML file to automate the tests, a question came to my mind:

Should the test_job run all the tests each time the pipeline is launched? Is there any best practice or convention regarding this?

Context

The project I’m working on is huge, and nothing about continuous integration or continuous deployment has been implemented yet. I am starting from scratch.

I thought the best way to start is by creating some test files, configuring the YAML file, and let some pipelines run so I could learn the basics and have something to start with.

Then I realized how big the project was, and how long it would take to run all the tests. It is simply not an option to run all of them each time a dev push into the repository.

Research Directions

Separating tests into test suites might resolve this problem; however, how can I know which test suite to launch?

I thought about two options:

1. Mapping-Based Approach:

   • Check the files that have been changed by the commits and use a mapping to know which test suite covers these specific files.

   • The problem is this solution is not dynamic at all, and the pipeline would not be able to launch freshly added files or tests if they’re not in the mapping. Additionally, it feels like bad practice to hard-code a mapping into the configuration.

2. Coverage-Based Approach:

   • Check the files that have been changed and use the coverage information of each test suite to determine which test suite covers the files from the commit.

   • The challenge here is that I have no experience with code coverage tools and am unsure if the effort put into learning them would resolve the problem.

There should be more options and potentially more issues with my current approaches. I would be pleased if you could enlighten me about them.

I believe this problem is common for large projects, but I haven’t been able to find resources or guidance to help me. How would you do it ?