Bigger tests than code?

It depends.

Sometimes the test will be more complex.

Sometimes the code will be more complex.

Refactoring to Simplify Code and Test Cases

It may be worth considering some simplifications and refactoring if things are getting exponential on you.

I have worked with several maintenance projects that were littered with God classes. These tended to be much more manageable when refactored into smaller classes.

Some classes are not minimal and a search for repeated code, and refactoring to eliminate repetition has the potential to reduce your test surface significantly.

Black Box vs. Glass Box Testing

One issue that arises is that of TDD methodology. When the user story changes, that produces more tests. On initial execution, they fail, and more code is written to make them pass. When they pass, the user story is implemented and we ship. Naturally, in real life, things might not be this pure. However, I guess it might be worth asking how much does the user story elaborate the test cases? Are the test cases functional, black box testing based on exercising the valid and invalid inputs?

It sounds from your description that some of the test cases are aimed at statement coverage. I like statement coverage, but there are several additional kinds of coverage as well. There are methods to optimize coverage test cases that you should perhaps consider if you find that part of your high test case count is due to brute forcing things to get through to every line.

Code Coverage Types

The following link illustrates with some cool graphics several types of coverage. I don’t endorse or have an opinion about their product, just their diagram.

http://www.atollic.com/index.php/trueanalyzer/types-of-code-coverage-analysis

In their example, they show modified condition/decision coverage and determine that for the code they show, four test cases both execute every statement, and cause the branching into the if statement for every kind of condition that might occur. Discovery of test cases when the decision criteria are nested comparisons like

if (a < 5) // d1
   if (b < 10) //d2
      a = 7;

if ((a + b) < 8) //d3
   c = 5;

can be trickier to analyze for test case values, but you know you will need a test case with a < 5 and a >=5 from the very first if, and also b < 10 and b >= b from the second if.

Because in some paths a will be the input, and others a will be a modified value (a=7), you may need to use a technique called symbolic execution where the third if is analyzed in terms of a’ and a”. To execute c = 5, you will need a test case where the condition is satisfied. You will also want to run a test where c is never assigned.

A list of test cases that I think covers everything is:

T1: a=4, b=9 => a'=4, a''=7, covers d1=true, d2=true, d3=false
T2: a=5, b=9, covers d1=false
T3: a=4, b=10, covers d2= true

But how do we get d3 covered?
Assigning a in the second if statement makes the analysis more complicated unless we do this with it.

((a' + b) < 8), ((a'' + b) < 8)

((a' + b) < 8), ((7 + b) < 8)  // Remember a'' = 7

((a' + b) < 8), (b < 1)  // Remember a'' = 7 

T4: a=1, b=0, covers d3=true

Symbolic execution is a test related technique that you can use to reduce the test cases needed to perform decision coverage. It is not an easy read, but a classic paper on the subject is:

http://www.cs.uiuc.edu/class/sp11/cs477/king76symbolicexecution.pdf

Apologies in advance if this diverts too far from your original question. I hope it will be of some use.

This may indeed happen when you write lots of unit tests and do a lot of TDD. When you come to a point where the test suite hinders you in making changes to a specific function in your code, then your test suite most probably contains too many (similar-looking) calls to this same function of your code. That’s almost always a sign for your test suite not beeing “DRY” any more. My suggestions to avoid that::

• refactor your test suite regularly to make it more DRY. Bundle your function calls to the code if they are looking similar in a central function, bundle the test data creation if the test data is reused for more than one test, write supporting functions to allow the comparison of complex objects with the expected values in less lines of code, and so on.
• write parameterized, data driven tests. Most modern unit test frameworks allow you to write a test function once and supply several sets of test data to it. If your framework does not have direct support for this technique, you can easily build it manually.

It may also be a good idea to study some “classic” test theory (see, for example, here) how to systematically design as few test cases as possible to achieve a high coverage.

I worked in a Java project in which we had more than 90% code coverage. The code of the unit tests was definitely more than the code of the project (I would reckon about 20 % more).

On the other hand we had very few bugs.

the test suite ends up being larger than the code itself.

Do my observations match that of everyone else?

• If you mean that the test suite as a whole has more LoC than all your application’s code, then no, I haven’t experienced that. Maybe because I’ve worked on projects with lots of unit tests, not many integration tests, and zero automated UI tests. The UI can really be huge in terms of lines of code, so I guess you really have to have comprehensive UI tests in order for the size of the test code base to exceed the total size of the production code base.

• If you mean your average test fixture has more LoC than the module it tests, it’s possible, especially if your production code has a lot of short, public methods.
  However, it could also mean that you’re testing too many things. Stuff like accessors and code constructs that contain no logic are generally considered not worth testing.

the larger the test suite, the harder it is to maintain the test
suite.

I guess there’s not a lot you can do about that mathematical fact. You can’t expect your test suite to come at no cost, and the cost will naturally increase with its size. There are things you can do about overly fragile tests though. See http://xunitpatterns.com/Fragile%20Test.html.

Having more code in the test for a given function is a sign that the function/method doesn’t stick to the single responsibility principle.

This means that in time the code becomes more convoluted and the tests need to grow in size exponentially to accommodate all the code paths and possible parameter combinations.

Most anti-patterns out there violate the single responsibility principle so it’s the first thing that goes out the window in a typical project. This leads to the situation that you mentioned.

A solution to this is to write concise tests for a given routine before writing the routine. If the tests are too complicated it means it’s time for refactoring.