I’m currently trying to unit test a behavior which, in some very particular cases, returns B instead of A. It may depend on the value of 3 different attributes for example.
How should I test this?
- Test only the specific case where the behavior is true
- Test the specific case and one other random combination of values for which the behavior isn’t true
- Test the specific case and all possible combinations
The last option seems like a maintenance nightmare as any new attribute or value for an attribute would require massive updates in all tests on behaviors that use these, but at least you would have 100% coverage of the behavior.
To give an example, we might want to know if a person is eligible for free tickles. To be eligible, you need to belong to category A, D and F, be more than 20 years old, be a lefty and have red hair.
Should I just test that “categories A or D or F + > 20yo + lefty + red hair” gives eligibility? Should I also test some random combination such as “E, 25yo, lefty, blond” is not flagged as eligible? Should I test absolutely every possible combination or category, below/above 20yo, hand and hair, one being true, all the others being false?
2
For a long boolean expression you generally don’t test every possible combination, but you test that each individual parameter has the appropriate effect. For example, given the following expression to test:
(A or B or C) and D and E
Test A by setting B and C false, and D and E true. Toggling A should now toggle the result of the entire expression.
Test B by setting A and C false, and D and E true. Toggling B should now toggle the result of the entire expression.
Likewise for the other variables. You set each one up so you know it’s the variable under test that’s changing the result.
That gives you 5 tests (O(n)) instead of 32 (O(2n)), but still provides good coverage.
A lot depends on what will be riding on the software.
If this is your basic Web app or your basic iPhone game, you probably don’t have to do THAT much.
If human lives will be riding on it, you darned well better believe you need to test the holy crap out of it.
The Therac 25 story should be well known by now. Several people died as a direct result of its software defect.
The Uwatec Aladin Air X Nitrox dive computer story is not as well known. Five divers were bent into pretzels, suffering major permanent neurological injuries, by its software defect. (Full Disclosure: I know one of the injured divers involved, slightly.)
If the number of possible combinations is reasonably small, you can test all of them. But if it is not the case and it is very difficult to maintain tests that cover all possible inputs, I would recommend testing only the most typical cases and edge cases(if there are any).