I was working with a team on a web-based software. We have written ~70-80 Selenium test for the product. Mainly happy-path tests with some regression ones for bugs reported by users.
It was the first time when our company did automatic UI end-to-end testing and it seems quite helpful. We have some success stories as well, like catching a serious regression in a library we used (it is developed by another team in the company).
Anyway, writing these tests needs a lot of time, partly because we didn’t have much experience with neither Selenium nor automatic UI testing.
After a few months of development my team and the product owner got the idea that we should make statistics about how much time writing and maintaining these tests took. (We usually made separate JIRA subtasks for Selenium tests so we could get a rough number.)
I’m afraid that this number alone could make a bad impression about testing. How can we produce a number to pair with this time to show benefits of testing as well, so we could decide whether it’s worth it (or not)?
We can dig JIRA for bugs which was caught by these tests but summarizing their logged work times does not seem good enough. Fixing a bug caught by automatic tests takes less time than fixing a released one reported by users and there are other benefits, for example, less interruptions for developers. Furthermore, catching a bug during a sprint gives faster feedback to the developers and usually we haven’t created separate JIRA subtask for these.
It is a good idea to summarize these time values at all?
1
Number of bugs caught or fixed has a very bad reputation as a software metric (it is no metric at all). Just Google it and you will find some horror stories.
Summarizing these time values indeed won’t tell much. What you want to tell your product owner alongside with the statistics is the following:
- You were working with a new tool, new methodology, you have set up a new configuration to run the tests. That means that you spent a significant amount of time learning, which is an investment. Because of this as you get more and more of an expert on this, the time you spend on writing this tests are likely to decrease in the futture.
- I assume you and your team have a good opinion about these tests, and you feel much safer releasing your product this way. Make sure your PO understands that, you are the experts after all. Emphasize that a bug caught before release is cheaper to fix by an order of magnitude then a bug caught after release.
- If I understand correctly, you are using scrum. If you think that you have a good way measuring the teams throughput (in userstory points or whatever), you can suggest to make statistics about how it grows in the long run because of the tests.
Final thoughts: for the UI tests to work effectively, you should have even more unit and integration tests. You can read abut this on Martin Fowler’s page.