Initial situation:
We are developing a software platform where products can be generated by configuring different components and adding additional ones. The platform is developed several years by a little developer team. Now the platform is more and more used, we get new developers as well as testers. We are currently developing a concept for developer. Cause is, that we do not have much developer tests for our components.
The tester team uses automated UI testing tools to test the products emitted by the platform.
Our goal:
We want to motivate our team members to write more tests to achive
- shorten feedback cycles
- reduce bug count for new features
- reduce regressions
The question:
We want to begin figuring out brittle components. After we have found them, we want to write integration tests for the component which describes expected features. Now we have the following problem: We can test the component for itself and mock all dependencies to other components. We can test the component with all directly depend components. We can test the component with the completely dependency structure of it – so its dependencies and their dependencies… We can test the component in the context of different product configurations.
Which one would be the best to achieve our goals and at the same time minimize the effort needed by our team members to configure the tests?
You will probably need some of both.
Integration tests are against real dependencies. They are usually easier to write, but require more time and resources to run. Some bugs will only be found by integration tests, because you will be affected by real bugs of the dependencies and you’ll never think of the same bugs when mocking. They also test all the components involved, so few tests for the downstream components will give you a lot of coverage.
Unit tests are against mock. Writing the mocks takes time, but the tests than run faster and it is the only way to test robustness against various failures, because you can’t make the real dependency fail in whatever way you think of. They are also much easier to debug on.
I suggest you start with integration tests. Create some scripts to set up all necessary test data so it is easy to run. Start with big, realistic setups. That should help you identify the brittle components. You can than create smaller setups to narrow down sources of the problems. Than create mocks and unit tests mainly as tool to make debugging the faults you found in integration tests easier. And keep them around as regression tests.
2