My first IT job was as a manual tester on an application. Within two months, I moved to an automated front-end tester. Now, on my app work, I write Unit tests to test my code. In my short career, I’ve pretty much been all over the app testing horizon.

The reason I bring that up is because every type of testing has value. On the first app with my first employer, we had an app with 670k lines of code, and only moderate Unit Testing coverage, which pretty much justified that job. That being said, a fair number of bugs we discovered and documented had a Unit Test created for them. Contrast to my current job; a project I am currently working on has < 2k lines of code, yet about 80% unit test coverage. Our testers haven’t yet pounded away on it, yet I have a feeling that when they do, my team will still have our share of suprises. It’s inevitable.

The problem with any type of testing is, it can never be 100% complete. You will asymptotically close on 100% coverage through any venue; the closer you get to a fully testable application, the closer you are incapable of getting to a fully-testable application. That thought in mind, I’ve grown to adopt the mindset of, ‘what does a particular type of testing do to ensure the quality of my app?’

Unit Testing: Ensures that internal dependencies and complex interactions work as designed.

Automated Front-End Testing: Regressively ensures that recent changes don’t break known-good features

Load/Performance Testing: Discovers issues that arise from large numbers of concurrent users.*

Manual Front-End Testing: Discovers issues that arise as a result of emergent behavior.

I have actually found bugs from executing Load Testing. Actually, you’d be suprised what you can find with a well-formed Load/Performance Test.

Answer to original question: both ‘camps’ make the same argument that some of us have encountered through blood, sweat, and tears: any sort of testing has its limits. However, those two camps you encountered are mutually wrong, because to ensure quality, all testing is necessary testing.

I think the controversy comes from resources. Not everyone has resources to perform all types of testing, yet I think we all know at some level that it would really help if we did. I see the problem of this controversy is less one of technology, and more one of the human element.

No kittens were harmed in the acquisition of this experience.

I try to take a balanced approach. On my personal projects, when company policy doesn’t mandate tests, I don’t test everything, but I will add unit tests the second time I fix a similar bug. Complex areas of my code end up with very good coverage, and the rest of the code is hardly covered at all. Perhaps that makes me a neutral party, if such a thing is possible in such a polarized topic.

I see the differences as simple personality conflicts. Some methods work better for some people. Testers like the methodical structure being added, and consider the time it takes to write and maintain tests as offset by productivity gains. The manual double checkers have good instincts and don’t tend to find many additional bugs with unit testing, and therefore consider the time it takes to write and maintain tests as not adding much value, especially since they tend to manually double check anyway.

I think the basic issue is that unit testing is an often-poorly-understood tradeoff. I write a lot of unit tests, but not for everything. There are many things I don’t think should have unit tests.

The basic thing you have to understand is that unit tests are regression tests. If a test fails, it does not mean there is a bug, and if a test passes it does not mean there isn’t a bug. If a test that was passing is now failing, it means, simply, that something changed. It’s an invitation to look and make sure that the change is correct and the test case was wrong, but it is an invitation only. Additionally I have found that there are often many unit tests which get coded poorly and so they break in unexpected ways and one has to follow these back and find out why. When the answer is, “We tested for the English-language OS-level message and we got a localized message from the OS in German” the correct response may, in fact be, to just get rid of that unit test since we can show it is not testing what it is supposed to test and there may not be an easy way to fix it.

In general, the more carefully spec’d out a portion of the code is, and the tighter the contract, the more important it is to unit test. Such test failures will often show you contract violations rather than bugs. Portions of the code which are not well spec’d out, subject to frequent change, and not part of a public API do not need as many (of in some cases any) unit tests. Manual testing there is ideal because you get real user feedback and automated tests will give you many false positives and false negatives.

In LedgerSMB we aggressively test things like number formatting and parsing, heavily test (but not as heavily) our database routines to varying degrees depending on, of course, how well defined the desired behavior is and how complex the code is. Something like a cash-basis income statement or a 1099 report gets a lot of testing. Something like saving a customer record gets light testing. The user interface layer gets light testing which amounts to “got an HTTP success and did not get an error page from the application” and test coverage there is unlikely to ever be complete.

But let me give the other side, namely LedgerSMB before the unit tests and after the fork. We forked from a codebase (of what was then a successful open source project!) which had no unit tests, and where most of the code cannot be tested because it relies on sloppy scoping. When we started adding unit tests the first ones we added were something simple: rounding numbers (keep in mind this is an accounting and ERP application, where rounding numbers is important). We got significant numbers of failures and we spent a fair bit of time correcting those failures. That single experience taught me that not to trust code which has no unit tests at all.

Unit tests are an important tool but “you must write unit tests for everything” is counterproductive. Similarly a lack of unit tests is an indication that there is a lack of, if you will, contract enforcement, and that behavior of some parts of the codebase might be very ill-defined. In the end, the tradeoff of unit testing a given portion of the code needs to be better understood and articulated, and the tradeoff is really that of rigidity in contract. There are many places where you want to test and many places where testing just gets in the way of making desired changes over development process.

The people who believe in unit testing fall into two camps. The “Having good coverage saved my ass from a bug that would have been a show stopper if that test wasn’t there” camp and the “Everyone says Unit Testing is a good thing and so I say it is too.” (Usually the second camp is paying lip service and not doing TDD until one day they decide to really do it and inevitably transition to the first camp).

The people who don’t believe in unit testing see more detriment (extra time writing and maintaining unit tests) than benefit (possibility of catching bugs early).

What they fail to realize is the freedom TDD gives you to maintain/update/change your code. Knowing that you can verify that the change you made didn’t create an unwanted side effect by running your unit test suite allows you to make changes more rapidly. In essence you’re paying an up front cost to build the tools that will allow you to go faster as the project grows.

So, why different testing methods are so contradicting topic?

What’s the old joke? “The only people who don’t love it are those who haven’t tried it.”

I don’t mean to be snarky, but I don’t know of anyone who worked in an environment where there was ubiquitous, small, isolated, speedy, repeatable, maintained automated tests and then argue against their utility in non-throwaway code.

That said, there are many developers who have worked in environments where one or more of those criteria failed. Tests suck when they’re not trustworthy. They suck when they take forever. They suck when they’re fragile. There are tons of ways to say you’re doing unit tests, and not actually do unit testing.

So yeah, many smart people will argue against what they know to be unit testing, which is a huge waste of time. Mostly though, the arguments I get against unit testing are from older developers than even me – “we’ve made quality software for decades without unit tests”. They fail to see the costs needed to achieve that quality, or how much more quality software they could have made by spending less time doing integration and bug hunting.

To understand where the two wildly different viewpoints come from, you need to have some understanding of history.

The concept of unit testing originated in the Smalltalk community. There are a lot of differences between Smalltalk and the languages that most people build large projects with these days, but the relevant one here is that Smalltalk is dynamically typed. And specifically (so that this doesn’t end up turning into a static-vs-dynamic flamefest, which is not the intention here,) it has no compiler capable of verifying type correctness prior to execution. This is the primary problem that unit testing was originally created to solve: catching details like type errors that the compiler would have caught for you, if you’d been able to use one. And that makes a lot of sense, when you don’t have a compiler available.

The problem is, a bit later down the line, a bunch of Java folks (who did have a compiler and a strict type system available) heard about the idea. Only they didn’t really understand it, most likely because the problem it’s designed to solve does not exist in Java. They thought, “oh, look at this! We can test our software’s functionality before we use it, and that will help catch bugs!” even though the type of bugs that unit testing was invented to catch don’t happen in Java.

Java being very popular, the idea got picked up and spread quickly, and the claims about it continued growing more and more grandiose, until eventually we ended up with ridiculous things like Uncle Bob claiming that, if you write all your code with a proper testing methodology:

think about what would happen if you walked in a room full of people
working this way. Pick any random person at any random time. A minute
ago, all their code worked.

Let me repeat that: A minute ago all their code worked! And it doesn’t
matter who you pick, and it doesn’t matter when you pick. A minute ago
all their code worked!

Just in case you didn’t catch the claim there, unit testing is supposed to be a magical thing that will make all your code work!

This is complete nonsense, of course. If all of your tests pass, that means that all of your tests pass, nothing more, nothing less. And specifically, it does not mean that:

• your tests correctly describe what the code is supposed to be doing in the first place
• your tests cover every possible error case
• your tests, being code too, are free from errors themselves. (Follow that one to its logical conclusion and you get stuck in infinite recursion. It’s tests all the way down!)

Also, bear in mind that your unit tests only cover the things you thought to write tests for. These are, of course, the things that were on your mind when you were writing them, areas where you anticipated would likely cause problems. These are areas you’re paying attention to, which you’re likely to get right in the first place. As any experienced developer will tell you, the majority of bugs (not all, of course, but a good majority) come from cases you never anticipated, which is why they turn out buggy. Tests are far less useful there.

There’s another serious problem with unit testing, one of those issues that you don’t really notice as a problem until your project gets really big and then it’s too late, which is that changing requirements requires changing code, which breaks your tests. Sometimes it can break a non-trivial fraction of your tests, like 10% of them. And if you follow Uncle Bob’s advice and write “every year… thousands of tests” like he exults about, eventually you have a whole lot of tests, enough that 10% of that whole lot is still a whole lot.

And the thing is, you just intentionally changed something. Unit testing is supposed to help you not accidentally make a change that breaks something, but you just changed something intentionally. And now you have hundreds or thousands (or more) of failing tests screaming at you, and you have to go through each one and individually, manually verify whether or not that test is still valid according to the new requirements. (And for the ones that aren’t, you might well have to write new tests to replace them!)

And the interesting thing is, long before Uncle Bob started making his ridiculous claims about unit tests magically making all your code work, long before unit testing spread to Java and from there to the rest of the programming world, long before Java even existed in the first place, the whole concept was already known to be a silly idea by people who actually understood programming!

It is now two decades since it was pointed out that program testing
may convincingly demonstrate the presence of bugs, but can never
demonstrate their absence. After quoting this well-publicized remark
devoutly, the software engineer returns to the order of the day and
continues to refine his testing strategies, just like the alchemist of
yore, who continued to refine his chrysocosmic purifications.

— Edsger W. Djikstra. (Written in 1988, so it’s now closer to
4.5 decades.)

The reason why there are two wildly different viewpoints out there regarding unit testing is that some people have enough perspective, and enough knowledge of history and/or of the fundamentals of computer science and engineering, to understand these principles and see the basic truth behind all the snake oil.

And some people don’t. What they see is that on some small project they’ve been working on, at some point they happened to catch a nasty bug one time with an automated test. Or they know someone who did. Personal experiences loom large in our memory–that’s basic human psychology. That’s the thing that they really remember, and so they think it has to be some wonderful, magical thing, logic and reason notwithstanding.

They hold to the concept of unit testing with dogmatic fervor, and like any dogma, its Absolute Truth is necessarily stronger than any fact. Just look at how often TDD proponents pull out the old “if you don’t like it/if it doesn’t work for you, you must be doing it wrong” argument when confronted with the facts. That’s not a logical argument; that’s an emotional attack meant to shut down logical discussion. And it works far too well (again, basic human psychology,) which is why people keep using it.

Unit test is a good alternative

Perhaps unit testing is hard or costly, but it is a good practice that has been widely embraced by our profession for having many advantages. Unit test:

• Brings the discovery of defects to the developer sooner with less visibility of those defects to others within the organization, let alone customers.
• Make developers more responsible for testing vs. a loosely defined developer test role.
• Permits whitebox/glassbox testing in ways only developers can perform.
• Broadens the base and diversity of people who contribute to the overall test automation used in the product.
• Supports continuous integration (CI) by permitting newly introduced defects to be automatically detected at the earliest opportunity.

On this last point, I recently saw a case where a driver developer made a change that had positive impact on the driver, but components from other groups had widespread failures. Without a pool of unit tests that ran automatically, the breakage would have been detected much later.

Software Development Thrives on Alternatives and Tools

Agile development is full of alternatives:

• Iterative/incremental vs. waterfall or spiral software life cycle models.
• Pair programming vs. inspections/code reviews.
• Developer gathered stories vs. system engineer driven formal requirements.
• More developers who unit test vs. more functional / regression testers.

As a developer embracing unit test you move against the tide of applying industrial revolution concepts like division of labor to software and instead embrace the change in which developers have a richer and more complete involvement with crafting their deliverable. This is also true when a developer helps annotate user stories or acceptance criteria in collaboration with customers.

Evolution and Complementary Options

Often, questions on SE Programmers question the value of broadly accepted practices. We are free to develop options by both evolving our tools and introducing new techniques. Source control tools evolved significantly from SCCS to CVS to SVN to Git. Software testing has as well, with test runners, assertion based testing often in conjunction with constant integration servers being one step along the way toward things that will be even more effective.

Other options that complement the benefits from unit test include:

• Static checkers.
• Code coverage tools.
• Code inspections and reviews.