Example:
A certain test is run five times on the current build, to compare it to release 1.0, where the same test was also run five times.
Build 1.4
22 ms, 26 ms, 23 ms, 25 ms, 20 ms
Release 1.0
15 ms, 18 ms, 16 ms, 20 ms, 17 ms
Question
The requirement is that Build 1.4 is no worse than Release 1.0.
How would I test that? I’ve seen different methods used, including simple average comparison and statistical T-Tests (assuming normal distributions).
None of the tools, i.e. testing frameworks and CI systems, I’ve come across provide anything to help with calculating and passing/failing such tests. Why? Performance testing seems to be popular, so how are people approving or rejecting them?
Often it’s way too impractical or even impossible to go in and manually determine acceptable ranges or distributions for every test.
4
Performance testing seems to be popular
It’s popular in theory. In practice, I’ve seen only a handful of automated performance tests done and they were done haphazardly.
How would I test that Build 1.4 is a pass/fail for this test?
You define a pass/fail criteria. If you need the tests to be consistent, define some failure criteria and measure. If you need the tests to be at least X fast (where X is ideally some metric gathered by usability testing) then do that. If not X worse than last build (not recommended in automated tests, since it requires memory of past builds) then do that.
It is trivial to write up helper code to deal with measurements and/or statistics. But you still need to define the criteria, just like any other test. It’s like asking to automatically define pass/fail criteria for business rules – computers do a poor job at that since all they know about business rules is what you tell them.
4
The answer to your question is highly case-specific. You already seem to be aware of various performance testing techniques.
You must determine what your expected measurements are. It may be acceptable to start by saying “our expected measurement is the current measurement” and then just track changes over time. If it’s “impractical or even impossible” to get an expected value then I don’t see how the performance measurement would be meaningful.
1
I can’t promise that this is optimal, only that it’s how I have been handling performance tests with moderate success.
Fairly substantial “real-world” integration tests get run under a timer, N repeats per test. Stash the results in a database. Periodically a set of results gets tagged as a waterline and subsequent runs are compared to that one, e.g. for a release, or when the testing hardware changes, or when we’ve implemented something cunning and everything runs faster.
The magic number used for comparison is the minimum of the results set. A fail occurs if it is greater than some fixed number (half a second or so) and is more than k% slower than the current waterline.
The minimum time seems to be pretty stable, but because the performance tests are run on a windows machine the worst case time jumps all over the shop. Periodically the test run collides with an antivirus scan or windows updates for instance. There’s consequently an automated retry failed tests step which runs at the end to try to cut out some difficult to reproduce failures.
The manual tuning involved in the k% and the fixed time is unfortunate. I suspect a stronger solution involves looking for statistically significant trends and triggering a failure based on a probabilistic metric. Still, the above is simple yet still catches performance regressions.