When writing a unit test for a scenario believed to be already covered, ie, the first run of the test would be green, what is a good guideline to ensure that it is in fact testing the proper test case, and not a test that ‘always’ passes, or passes for reasons that have nothing to do with the test case being considered ?
I believe the best practice is to change your class under test, so that it fails first, and then remove the intentional bug to see if it turns green, however that explanation is too vague. At an extreme, I could simply make my constructor throw an exception to get a red test, then remove the throw to make it green. I know the breaking change should be something ‘close’ to the scenario under test, but am not sure how to word it as a proper guideline that my juniors could follow.
1
what is a good guideline to ensure that it is in fact testing the proper test case
This falls back to the old “who tests the tests?” line of thinking.
And in general, nobody can test the tests. Unit tests have proven themselves so successful in part because they are small enough that they can be implemented without error. A good guideline is that another developer can look at the test and say “yes, that appears to test what it says it does”.
That is enough to catch the majority of issues. The rest is part of a good layered approach to software quality. Even if the test is wrong, QA or user acceptance testing can help catch things that escape the test. That reduces the risk of a bug escaping into production to an acceptable amount.
4
Even though it is right that you should not start testing your tests the ability of a test to fail is crucial. A test which cannot fail is not only useless – it also indicates that you …
- made logical mistakes
- just wanted to do some quick testing without thinking it through or
- are testing the language (e.g.
unsigned>= 0) or general logic (true || false)
If you already have the code implemented please don’t make the test fail by crashing the program. The following test will allways fail, even though it is a tautology.
#include <gtest/gtest.h>
TEST(test,testcase) {
throw 1;
EXPECT_TRUE(true);
}
int main(int argc, char ** argv) {
::testing::InitGoogleTest(&argc,argv);
return RUN_ALL_TESTS();
}
(Maybe there are test frameworks in which this test would still pass, but there is at least one in which it gives you false security.)
Instead just write a quick “empty implementation” which has the same interface and simply does nothing. You then – when all tests failed – switch out the dummy with the real implementation.