I recently learned about unit tests and TDD and have started going back to add unit tests to my code (and thank god, it’s proven that some things I’ve written are much more broken than I thought).
That being said, I’m having a hard time figuring out how much (or little) should go into each test. For example, consider a class I wrote with several methods. Right now I do something like
class test_myclass(unittest.TestCase):
def setUp(self):
...
def test_func1(self):
...
def test_func2(self):
...
etc
def tearDown(self):
...
But what if each (or at least some) method can fail in a lot of different ways, or has a lot of edge cases? Then I’m stuck doing something like
def test_func1(self):
self.assertEqual(func1(working_input), expected_value)
self.assertRaises(Some_error, func1, args)
self.assertRaises(Some_other_error, func1, args)
etc
And it could get pretty hairy, and rather unclear where things are failing. So I thought I might break it up more, and devote a class to each function (that has enough test cases to be worth it) but that seems like a lot of work, and more time than I should probably be spending on testing.
I also considered writing multiple tests for a given method, such as
def test_func1(self):
self.assertEqual(func1(working_input), expected_val)
def test_func1_bad_input_1(self):
self.assertRaises(error, func1, bad_args)
def test_func1_bad_input_2(self):
self.assertRaises(error2, func1, bad_args_2)
...
And this seems cleaner than a unit testing class per method, but still seems like it might be to little in any given test.
How much testing is too much?
While my example was given in python, I’m open to language agnostic answers
In an ideal world, a single unit test has only one thing that can cause it to fail, and that one thing can only cause that test to fail.
I’m not sure I’ve seen that ever be true though… A more attainable approach is that a single unit test tests a single path though a function. The concept of code coverage is helpful here, but the gist of it is that any decision Your code makes represents (at least) a different path. Some paths may be too difficult (or actually impossible) to unit test, but that is the level of granularity to work towards.
The benefit is that when a test fails, you spend less time and effort figuring out why.
And yes, it seems like a bit more work, but once you’re used to it, the tests are quick and easy to write – and they’ll save you tons of time.
You struggle since you are not doing TDD – you are currently adding unit tests afterwards instead of before. Using TDD “by the book” means: work with extremely short cycles of
- think of the next requirement/change you want to implement
- write a test for exactly that requirement
- run the test (gives you typically “red”)
- implement a change for the requirement
- run test again (hopefully green, fix the code if not “green”)
- refactor (code and tests)
When you work this way, the answer to your question “How much testing is too much?” becomes obvious: you write exactly the amount of tests you need to be sure the next feature or change to your class or function work as intended – no more, no less.
TDD is less a testing, but more an implementation technique. Writing the tests and implementing your code is completely interwoven. When you have implemented a feature or change, you should have already written most of the tests beforehand you need for assuring the feature works as intented – so when the function is “ready”, there are no tests to write any more. When you feel you might have missed some edge cases, and you are not sure your function handles them correctly, then add some extra tests afterwards. But that also may be a sign of implementing already too many things at once – better just add a test and the code for the “happy path” first, afterwards add the test and the code dealing with those edge case – in exactly that order.
In a real-world scenario, working always that way may be too idealistic, but once you understand how TDD was meant, the question of “how much to tests to write” is much easier to answer.
Once you figured out how many tests you really have to write, the question how to break them up is much easier to answer: you break them up (in the refactoring step above) until you think the design and structure of the tests you have written so far is good enough. The SOLID principles are a good guideline not only for your production code, but also for your tests as well. So for example, as long as you have just one test for a function, a single test function is fine. When you discover you need more tests for the same function, there may be the point where you need a full class of tests just for a single function.
Unit tests give you value, and they create work. You want the best possible value for work.
If you write ten tests each checking ten items, instead of 100 tests each checking one item you usually safe some work. You still have the same test coverage. So if your tests work, you are fine. If they fail, and you ignore it, you are also fine.
In the case where you have a failing test and are actually fixing it, you have some work figuring out which item failed, then some work to figure out why it failed, and some more work to fix it. I’d say that usually the saving that you have from knowing which of 100 items failed vs. which group of ten items failed is minimal. It is usually more than offset by the time saving that you have while writing the tests.