The idea of TDD is to start with testing and work from that. Thus, to take your example of “A catalog needs to have a list of products” could be seen as having a test of “Check for products in catalog” and thus this is the first test. Now, what holds a catalog? What holds a product? Those are the next pieces and the idea is to get some bits and pieces put together that would be something like a ProductService that will be born from getting that first test to pass.

The idea of TDD is to start with a test and then write the code that makes that test pass as the first point. Unit tests are part of this yes, but you aren’t looking at the overall picture that is formed by starting with tests and then writing the code so that there aren’t blind spots at this point since there isn’t any code yet.

Test Driven Development Tutorial where slides 20-22 are the key ones. The idea is to know what the functionality should do as a result, write a test for it and then build a solution. The design part will vary as depending on what is required it may or may not be that simple to do. A key point is to use TDD from the start rather than try to introduce late into a project. If you start with tests first this can help and is likely worth noting in a sense. If you try to add the tests later, it becomes something that may be put off or delayed. The later slides may also be useful as well.

A main benefit of TDD is that by starting with the tests, you aren’t locked into a design initially. Thus, the idea is to build the tests and create the code that will pass those tests as a development methodology. A Big Design Up Front can cause problems as this gives the idea of locking things into place which makes the system being built to be less nimble in the end.

Robert Harvey added this in the comments which is worth stating in the answer:

Unfortunately I think that this is a common misconception about TDD:
you can’t grow a software architecture by just writing unit tests and making them pass. Writing unit tests does influence the design, but
it doesn’t create the design. You have to do that.

For what it’s worth, TDD helps me come to the best design more quickly than not doing TDD. I would probably come to the best design with or without it. But that time that I would have spent thinking it through and taking a few stabs at the code is spent writing tests instead. And it’s less time. For me. Not for everyone. And, even if it took the same amount of time, it would leave me with a suite of tests, so that refactoring would be safer, leading to even better code down the line.

How does it do it?

First, it encourages me to think about every class as a service to some client code. Better code comes from thinking about how the calling code wants to use the API rather than worrying about how the code itself should look.

Second, it stops me writing far too much cyclometic complexity into one method, while I’m thinking it out. Each extra path through a method will tend to double the number of tests I need to do. Sheer laziness dictates that after I’ve added too much logic, and I have to write 16 tests to add one condition, it’s time to pull some of it out into another method / class and test it separately.

It’s really that simple. It’s not a magic design tool.

I’m trying to wrap my head around TDD…
To illustrate, imagine these 3 requirements:

• A catalog needs to have a list of products
• The catalog should remember which products a user viewed

These requirements should be restated in human terms. Who wants to know which products the user previously viewed? The user? A salesperson?

• Users should be able to search for a product

How? By name? By brand? The first step in test-driven development is to define a test, for example:

browse to http://ourcompany.com
enter "cookie" in the product search box
page should show "chocolate-chip cookies" and "oatmeal cookies"

>

At this points, many books pull a magic rabbit out of a hat
and just dive into “Testing the ProductService”, but they don’t
explain how they came to the conclusion that there is a ProductService
in the first place.

If these are the only requirements, I certainly wouldn’t leap to create a ProductService. I might create a very simple web page with a static product list. That would work perfectly until you get to the requirements to add and delete products. At that point I might decide it is simplest to use a relational database and an ORM, and create a Product class mapped to a single table. Still no ProductService. Classes like ProductService will be created when and if they are needed. There may be multiple web requests that need to perform the same queries or updates. Then the ProductService class will be created to prevent code duplication.

In summary, TDD drives the code to be written. Design happens as you make implementation choices, and then refactor the code into classes to eliminate duplication and control dependencies. As you add code, you will need to create new classes to keep the code SOLID. But you don’t need to decide ahead of time that you will need a Product class and a ProductService class. You may find that life is perfectly fine with just a Product class.

Others may disagree, but to me many of the newer methodologies rely on the assumption that the developer is going to do most of what the older methodologies spelled out just out of habit or personal pride, that the developer is usually doing something that is fairly obvious to them, and the work is encapsulated in a clean language or the cleaner parts of a somewhat messy language so you can do all the test business.

Some examples where I have run into this in the past:

• Take a bunch of spec-work contractors and tell them their team is
  Agile and Test First. They often have no habit other than to work to
  spec and they have no concern over the quality of the work as long as
  it lasts long enough to finish the project.

• Try and do something new test first, spend much of your time ripping
  tests as you find various approaches and interfaces are crap.

• Code something low level and either get slapped for lack of coverage,
  or write a lot of tests that do not amount to much value because you
  cannot mock the underlying behaviors you are tied to.

• Any situation where you don’t have enough of the underlying mechanics in place ahead of time to add a testable feature without writing a bunch of underlying untestable bits first, like disk subsystem, or a tcpip level communication interface.

If you are doing TDD and it is working for your, good for you, but there are a lot of things (whole jobs, or stages of a project) out there where this just simply does not add value.

Your example sounds like you aren’t even to a design yet, so either you need to have an architecture conversation, or you are prototyping. You need to get through some of that first in my opinion.

I am convinced that TDD is a very valuable approach to the detailed design of the system – i.e. the APIs and the object model. However, to get to the point in a project where you would begin to use TDD, you need to have the big picture of the design already modeled in some fashion and you need to have the big picture of the architecture already modeled in some fashion. @user414076 paraphrases Robert Martin as having a design idea in mind, but not being married to it. Exactly. Conclusion – TDD is not the only design activity going on, it is how the details of the design get fleshed out. TDD must be preceded by other design activities and fit into an overall approach (such as Agile) that addresses how the overall design gets created and evolved.

FYI – two books I recommend on the topic that give tangible and realistic examples:

Growing Object-Oriented Software, Guided by Tests – explains and gives a full project example. This is a book about design, not testing. Testing is used as a means of specifying expected behavior during design activities.

test-driven development A Practical Guide – a slow and step-by-step walk through developing a complete, albeit small, app.

TTD drives design discovery by test failure, not success, therefore you can test unknowns and iteratively retest as unknowns are exposed ultimately leading to a complete harness of unit tests – a very nice thing to have for ongoing maintenance and a very difficult thing to try to retrofit after code is written/released.

For example, a requirement may be that input can be in several different formats, not all are known yet. Using TDD you would first write a test that verifies the appropriate output is supplied given any input format. Obviously this test will fail, so you write code to handle the known formats and retest. As the unknown formats are exposed through requirements gathering, new tests are written before the code is written, these also should fail. Then new code is written to support the new formats and all tests are rerun reducing the chance for regression.

It is also helpful to think of unit failure as “unfinished” code instead of “broken” code. TDD allows for unfinished units (expected failures), but reduces the occurrence of broken units (unexpected failures).

In the question it is stated:

… many books pull a magic rabbit out of a hat and just dive into “Testing the ProductService”, but they don’t explain how they came to the conclusion that there is a ProductService in the first place.

They came to that conclusion by thinking about how they were going to test this product. “What sort of product does this?” “Well, we could create a service”. “Ok, let’s write a test for such a service”

A functionality can have many design and TDD will not completely tell you which one is the best. Even, if tests will help you build more modular code, it can also lead you to build modules that suits tests requirement and not production reality. So you have to understand where you are going and how things should fit in the whole picture. Put otherwise, there are Functional and Non-Functional requirements, don’t forget the last one.

Concerning design I refer to Robert C. Martin books (Agile Development) but also Martin Fowler’s Patterns of Enterprise Application Architecture and Domain Driver Design. The later especially is very systematical in extracting the Entities and Relations out of the requirements.

Then, when you get a good feelings of the options available to you on how to manage those entities, you can feed you TDD approach.

Isn’t TDD intended to make me discover my design in the first place?

No.

How can you test something that you have not designed first?

To illustrate, imagine these 3 requirements:

• A catalog needs to have a list of products
• The catalog should remember which products a user viewed
• Users should be able to search for a product

These are not requirements, these are definitions of data. I do not know what is the business of your software, but it is not likely that analysts speak that way.

You need to know what are the invariants of your system.

A requirement would be something like:

• A customer can order a certain quantity product, if there is enough of this product in stock.

So if this is the only requirement, you may have a class like:

public class Product {

  private int quantity;

  public Product(int initialQuantity) {
    this.quantity = initialQuantity;
  }

  public void order(int quantity) {
    // To be implemented.
  }

}

Then using TDD, you would write a test case before implementing the order() method.

public void ProductTest() {

    public void testCorrectOrder() {

        Product p = new Product(10);
        p.order(3);
        p.order(4);

    }

    @Expect(ProductOutOfStockException)
    public void testIncorrectOrder() {

        Product p = new Product(10);
        p.order(7);
        p.order(4);

    }

}

So the second test will fail, then you can implement the order() method the way you like.

You are quite correct TDD will result in a good implementation of a given design. It will not help your design process.

TDD helps a lot, however there is important part in software development. Developer should listen to the code that is being written. Refactoring is 3rd part in TDD cycle. This is the main step where developer should focus and think before go to next red test. Is there any duplication? Are SOLID principles applied? What about high cohesion and low coupling? What about names? Take a closer look at code that is emerging from the tests and see if there is something that needs to be changed, redesigned. Question the code and code will tell you, how it want’s to be designed.
I usually write sets of multiple tests, examine that list and create first simple design, it doesn’t need to be “final”, usually it’s not, because it’s changed when adding new tests. That’s where the design comes.