When writing (parameterized) tests for stateless functions, how do I determine the most ideal set of values to test with?
I’m writing a class with mathematical functions in C#, and I want to write unit tests for them. Today I want to write this function (really!):
int Clamp(int value, int lowerBound, int upperBound)
{
// TODO: Implement.
return 0;
}
As good TDD practice dictates, I’m writing the unit tests first and the implementation later. I can write a parameterized test for this stateless function, but have to decide on the best set of values to test with.
My current strategy for picking testing values is:
For each parameter, I can pick values from the whole set of 32-bit signed integers. I’m not using randomly generated values as that would make the test non-repeatable. To test the corner cases, I pick -1, 0, 1, but also the −2147483648 (minimum 32-bit signed integer) and 2147483647 (maximum 32-bit signed integer). I would also need to pick a ‘normal’ value, for example, 42, to make sure I’m not only testing corner cases. Is one normal value enough? And a negative one, let’s say -63. Did I miss any?
−2147483648, -63, -1, 0, 1, 42, 2147483647
If any of the three parameters can have any of just these seven values, I’ll end up with 343 tests for one function! Before I’m going to write 343 tests, I really have to ask:
Are these the right values to test with? In general, what strategy can I use to pick the most effective values to use?
4
You never said what Clamp() is supposed to do, so I’m assuming that it returns value, unless it is outside of the range, in which case it returns one of the two bounds.
I don’t see any reason to think that -1, 0, or 1 are corner cases. They may often be corner cases, but there’s no reason they’d act strangely in this function. If you want a ‘normal’ value, 42 or -63 works, but there is no need for both of them, unless you suspect that > and < don’t work properly on negative numbers in C#. (I don’t think you need to worry about that.)
So we could just use −2147483648, ‘a normal value’, and 2147483647. (We could even say that testing with the max/min integer values aren’t really necessary. Presumably, C# > and < work up to the minimum and maximum; there isn’t any danger of integer overflow.)
There are 6 permutations of 3 values, so we’re down to 6 testcases. 6 testcases is not much, and we can easily just write them down and use them, but we don’t know for certain that we’ve selected test cases that cover everything (all we’ve done so far is reduce the original set of test cases to something smaller).
If we want to be sure we’ve caught all the cases that matter, we could reduce the massively large set of input values (4 billion cubed) by partitioning them into equivalence classes. Then we only need 1 test per equivalence class, since the equivalence class would be defined as a set of inputs that all act alike.
The value of Clamp(a, b, c) depends on whether a is in the range, or above it, or below it. There should be 3 equivalence classes: [a < b and a < c], [a > b and a > c], and otherwise. The return value will be b, c, or a, respectively. This tells us not only what the tests should be, but how to write the code.
(There is one little thing that we haven’t run into: what if the lower bounds is higher than the upper bounds. What I said in the previous paragraph applies if the assumption I made up at the top is right, but not if it isn’t. It can be fixed easily, though, by swapping b and c or by returning Clamp(a, c, b) if b > c.)
4
Test Driven Development principles suggest that you test cases cover a range of conditions, not only testing over a range of valid input, but testing for handling of error conditions as well.
So first start testing how it handles invalid / valid input, to see if it correctly handles erroneous input,
That includes:
- Simple known good case (PASS)
- Simple known “bad” input (FAIL)
- Based on the underlying datatype, values at boundaries for that datatype (e.g. INT_MAX, INT_MIN, a string of maximum or minimal length including the empty string)
- With numerical values, testing a signed value
- Testing values over a complete range of input values, beyond what makes sense for the application usage. If you assumes the parameter is normally going to be between 1 and 10, how does it handle 1000000 or -10? This can be tricky to comprehensively test for floating point in particular.
The extensiveness of testing is determined on factors such as how widespread usage is (i.e. is it an one-off usage at the moment versus inclusion in a common library), how critical are failures? (user loses a casual game versus vehicle’s brakes cease to function)
If any of the three parameters can have any of just these seven values, I’ll end up with 343 tests for one function! Before I’m going to write 343 tests,
Don’t confused tests with test cases, test cases should be thought of as mere data.
You should write code that loops through those test cases that may be stored as external CSV text files, and see if they match the expected outcome. Break the cases into lists of “good” (expect to pass) values and “bad” (expect to fail) values and simply test whether the function behaves (PASS/FAIL) as expected give the combination of input data (all cases with valid input should pass, all cases with any invalid input should fail), without verifying the correctness of the output.
Then for a set or subset of interesting input values you can compare the output values to the expected output values, that are normally generated independently of the code being developed (e.g. for many simpler cases for numeric calculations a spreadsheet can be used), making sure to handle floating point comparisons sanely.
If you generate the test case data automatically, then the number of test cases only becomes a factor once the execution speed is an issue, an increasingly rare issue when testing at function test.
In the case of your example function, where for any input the output should be valid, then you can simple test the correctness of the output given the particular combination of input values.
It does help force you to limit the numbers of parameters to a function, which is normally a good thing as it limits the functions to do one thing well, and not become a ugly monster of special cases based on those additional values.
2