Software QA FYI - SQAFYI

Test Driven Development (TDD) Traps

By: Jakub Nabrdalik

Have you ever been in a situation, where a simple change of code broke a few hundred tests? Have you ever had the idea that tests slow you down, inhibit your creativity, make you afraid to change the code? If you had, it means you've entered the Dungeon-of-very-bad-tests, the world of things that should not be.

I've been there. I've built one myself. And it collapsed killing me in the process. I've learned my lesson. So here is the story of a dead man. Learn from my faults or be doomed to repeat them.

The story
Test Driven Development, like all good games in the world, is simple to learn, hard to master. I've started in 2005, when a brilliant guy named Piotr Szarwas, gave me the book "Test Driven Development: By Example" [1], and one task: create a framework.

Those were the old times, when the technology we were using had no frameworks at all, and we wanted a cool one, like Spring, with Inversion-of-Control, Object-Relational Mapping, Model-View-Controller and all the good things we knew about. And so we created a framework. Then we built a Content Management System on top of it. Then we created a bunch of dedicated applications for different clients, Internet shops and what-not, on top of those two. We were doing well. We had 3000+ tests for the framework, 3000+ tests for the CMS, and another few thousand for every dedicated application. We were looking at our work, and we were happy, safe, and secure. Those were good times.

And then, as our code base grew, we came to the point, where a simple anemic model we had, was not good enough anymore. I had not read the other important book of that time: "Domain Driven Design" [2], you see. I didn't know yet, that you could only get so far with an anemic model.

But we were safe. We had tons of tests. We could change anything.

Or so I thought.

I spent a week trying to introduce some changes in the architecture. Simple things really: moving methods around, switching collaborators, such things. Only to be overwhelmed by the number of tests I had to fix. That was TDD, I started my change with writing a test, and when I was finally done with the code under the test, I'd find another few hundred tests completely broken by my change. And when I got them fixed, introducing some more changes in the process, I'd find another few thousand broken. That was a butterfly effect, a chain reaction caused by a very small change.

It took me a week to figure out, that I'm not even half done in here. The refactoring had no visible end. And at no point my code base was stable, deployment-ready. I had my branch in the repository, one I've renamed "Lasciate ogni speranza, voi ch'entrate".

We had tons and tons of tests. Of very bad tests. Tests that would pour concrete over our code, so that we could do nothing.

The only real options were: either to leave it be, or delete all tests, and write everything from scratch again. I didn't want to work with the code if we were to go for the first option, and the management would not find financial rationale for the second. So I quit.

That was the Dungeon I built, only to find myself defeated by its monsters.

I went back to the book, and found everything I did wrong in there. Outlined. Marked out. How could I skip that? How could I not notice? Turns out, sometimes, you need to be of age and experience, to truly understand the things you learn.

Even the best of tools, when used poorly, can turn against you. And the easier the tool, the easier it seems to use it, the easier it is to fall into the trap of I-know-how-it-works thinking. And then BAM! You're gone.

The truth
Test Driven Development and tests, are two completely different things. Tests are only a byproduct of TDD, nothing more. What is the point of TDD? What does TDD brings? Why do we do TDD?

Because of those three reasons.

1. To find the best design, by putting ourselves into the user's shoes.
By starting with "how do I want to use it" way of thinking, we discover the most useful and friendly design. Always good, quite often that's the best design out there. Otherwise, what we get may be theoretically correct, but terribly hard to use.

Do you remember EJB 1 or 2? CORBA? Any technology designed by a committee, is going to be "correct", but overengineered and painful to use. And you don't want that.

The best design is discovered by the one who has to use it.

2. To manage our fear.
It takes balls, to make a ground change in a large code-base without tests, and say "it's done" without introducing bugs in the process, doesn't it? Well, the truth is, if you say "it's done", most of the time you are either ignorant, reckless, or just plain stupid. It's like with concurrency: everybody knows it, nobody can do it well.

Smart people are scared of such changes. Unless they have good tests, with high code coverage.

TDD allows to manage our fears, by giving us proof, that things work as they should. TDD gives us safety

3. To have fast feedback.

How long can you code, without running the app? How long can you code without knowing whether your code works as you think it should?

Feedback in tests is important. Less so for front-end programming, where you can just run the shit up, and see for yourselves. More for coding in the backend. Even more, if your technology stack requires compilation, deployment, and starting up.

Time is money, and I'd rather earn it, than wait for the deployment and click through my changes each time I make them.

And that's it. There are no more reasons for TDD whatsoever. We want Good Design, Safety, and fast Feedback. Good tests are those, which give us that [4] Ok, Iím lying a bit in here. Authors like Kent Beck and Tomasz Kaczanowski mention more reasons. For example: using TDD in communication (Behavior Driven Development and user stories are a great example). In my practice, though, Iíve found only those three to be of a great importance. Others were easily replaced with different (sometimes better) tools and methods than TDD. Your mileage may vary, so find out whatís best for you..

Bad tests?

All the other tests are bad.

The bad practice

So how does a typical, bad test, look like? The one I see over and over, in close to every project, created by somebody who has yet to learn how NOT to build an ugly dungeon, how not to pour concrete over your code base. The one I'd write myself in 2005.

This will be a Spock sample, written in groovy, testing a Grails controller. But don't worry if you don't know those technologies. I bet you'll understand what's going on in there without problems. Yes, it's that simple. I'll explain all the not-so-obvious parts.
def "should show outlet"() {
def outlet = OutletFactory.createAndSaveOutlet(merchant: merchant)
response.redirectUrl == null

So we have a controller. It's an outlet controller. And we have a test. What's wrong with this test?

The name of the test is "should show outlet". What should a test with such a name check? Whether we show the outlet, right? And what does it check? Whether we are redirected. Brilliant? Useless.

It's simple, but I see it all around. People forget, that we need to:

Trick 1: Verify the right thing

I bet that test was written after the code. Not in test-first fashion. But verifying the right thing is not enough. Let's have another example. Same controller, different expectation. The name is: "should create outlet insert command with valid params with new account"

Quite complex, isn't it? If you need an explanation, the name is wrong. But you don't know the domain, so let me put some light on it: when we give the controller good parameters, we want it to create a new OutletInsertCommand, and the account of that one, should be new.

Full article...

Other Resource

... to read more articles, visit

Test Driven Development (TDD) Traps