Why TDD is Bad (and How to Improve Your Process)

Well the title is more or less a click-bait because TDD is not really bad. So first of all I want to claim my point of view:

• I’m not against tests. Writing test cases is always an important step in software development process.
• I’m not against most of the practices in TDD. TDD employs many great practices, such as writing test cases before writing code, improving test coverage, keeping the unit small, these are all good.
• I’m not arguing about TDD is all bad. In some case it is good enough and effective, but most cases people using TDD in the wrong situations.

What I’m against is the fundamental concept of TDD: Test Driven. What we really need is a methodology that can produce high quality software within reasonable time and cost, and the concept of TDD will not lead us to our goal.

TDD, by Example

I was in a team where Agile and TDD were used. I have to admit that our team was doing a great TDD process. I learnt a lot from them, which is good. The TDD process I learnt was like this:

Steven: “As an example Bob and me will do a TDD chess game on our next back-end story APS-1042.”

(A few hours later…)

Steven: “Okay I’ ve created my first test case on branch feature/APS-1042. This test case expects our book count API to return 10 after 10 books are created. Now the test result is RED. Now Bob will write code to make the test GREEN again.”

Bob: “To demonstrate the minimal unit principle I will write exact code to pass this test case. My code will be only one line: return { count: 10 }. Now the test is GREEN, and its Steven’s turn to add more test cases.”

Steven: “I noticed that Bob was cheating, but this revealed that my test case didn’t cover the case that the API happens to return 10 and 10 only. I will add another test case, creating random number of books and make sure the result is equal to that random number.”

Bob: “Obviously my previous code did not work under the new test case. Realizing Steven was using random count, I have to count the books in the database and return him the correct count.”

Steven: “Okay now it looks all good, it’s time to add some useful code. I need to make sure that the count does not include books on loan.

……

Although exaggerate but this was the correct TDD workflow. Write test case, make test fail, then write code, make test pass, and repeat. Looks good. So what’s wrong with it?

TDD Focus on Passing Test, Not Correctness

People who trust in TDD would say, “My test cases are the specification. If all tests are passed, the correctness is guaranteed. ”

Unfortunately that is not true. Test cases are never equal to specification.

In simple cases such as “return the count of non-fiction books” it might be true because the feature can be directly described by test cases. But what about complicated features, for example, “make API calls against different servers (100+) then aggregate the result and return the count, but simultaneous API calls to the same server should be limited, for example, maximum 6 calls in 30s, and each server should be able to set its own limit”?

Since unit tests focus on implementation, it can’t be done until the implementation becomes clear. For complicated features as above, it is not clear how it should be done, what components should be involved, what functions should be created. Thus it is impossible to write test cases unless a clear design is made, and frequently, some proofs of concept are created.

But why don’t I just improve my proof of concept and make it ready for production? Isn’t easier and faster than creating test cases from scratch?

Someone would argue that “You should decompose the complicated feature into several small stories of 3–5 point size”. Yes you are correct. But please note the above example is nothing more than a simple rate limiting algorithm, modified to adapt multiple servers. In the real world we coded that feature with less than 20 lines of code, along with several auxiliary functions that handle cache, queue and others. We estimated it as a five point story and it was pretty precise. Also the functions are so closely related that dividing them into several stories will definitely increase communication cost.

So what would I do if practicing TDD on this example?

The best result I could think is like this.

1. I will do a quick meeting with my team to make some design decisions as the base of the test cases.
2. Then I will start writing test cases for the first iteration.
3. My partner will start coding to pass my test cases.
4. I will add more test cases.
5. My partner will find his code not suitable for my new test cases and some refactoring might be needed before he could write new code.
6. Repeat 4–5.

In this process, we may encounter the following risks.

• Since we only did a quick meeting we won’t be able to see everything required by this feature. We may have missed the cache, making wrong decision about queuing, using an improper rate control algorithm, and so on. All these design flaws will result the code to be “refactored”, or more precisely, rewritten.
• I could not see the big picture when creating test cases, so very likely I won’t be able to cover all the edge cases.
• My partner would treat my test cases as “specification” and only does what I asked him to do without thinking of what a rate limit algorithm should be. This will result in redundant code, low performance, and in most cases, code full of bugs.
• Since we need to run the test suite I have to mock any external dependencies, I may ignore some characteristics of the dependencies, for example, my test cases may fail to consider the expiration of cache.

I can imagine, we will get some working code at the end of day, but it could be nothing more than a piece of long, patched code. And we may have spent 1~2 weeks on it.

Solution

The root cause of this is, TDD makes people focus on passing tests, not correctness.

And passing tests is NEVER a key to good quality.

“Quality is made by design, not testing.”

I am a developer from the era when waterfall was still in the main stream of software engineering. We got experiences of making good software with waterfall. The most significant thing I learnt from waterfall was that software quality is assured in design phase, not in test phase.

Today most teams adapted TDD along with Agile, and what they actually did was dropping the design phase from the process and starting coding immediately.

Wrong. This is proven (by many failed projects) to be the worst way because later you will spend 10x times fixing bugs.

Think about the life cycle of TDD: write test cases → coding → refactoring. The “refactoring” here is exactly the extra cost fixing defects in coding / testing phases than in design phase.

So what should we do?

A design phase should be added before TDD begins. Since we only deal with one single feature, we don’t need several months in design phase, like in waterfall. Sit down with your team, discuss every perspective of the feature, divide it into components, design the architecture, decide the interface, and identify the potential risks. Its just like a “micro” design phase, and usually it takes only a couple of hours. Then you can start TDD or whatever.

Trust me, if you are a TDD fan, this will make your TDD life way easier.

And much less complains from QA team and operation team.

TDD is Time Consuming and Costly, in both Short Term and Long Term

In previous section we’ve already discussed why TDD is time consuming in short term: you have to spend significant time on refactoring and rewriting your code. But in the long term it will cost more time as well.

Remember, test cases are code, too. When feature changes, implementation will change as well, and many test cases will fail. At this time you will have to fix test cases as well, which usually takes as much as, or even more time than you change your implementation.

Please note that I did not mention the order at all. You can fix implementation then test cases, or in opposite order, it does not matter. Both approaches cost almost the same amount of time.

This problem exists as long as unit test exists, but more severe especially when the test cases are written during TDD. Since during TDD process people tend to focus on implementation, the test cases are more prone to change.

In fact, this is not a “bad” thing — test coverage is always good. But good coverage means more cost. And the goal of software engineering is to find a balance between cost, time and quality.

Solution

Instead of TDD, I prefer a similar approach called “BDD” (Behavior Driven Development). Instead of writing test cases for every function in the implementation, write test cases for the behaviors.

Of course the definition of “behavior” can be different: It can be story level behavior, making one test case for each component in the story; or it can be more granular, making test cases for all the actions such as API calls and DB queries. Choose what fits your project best.

The important thing here is, focus on the behavior, not the implementation. If a behavior consists of three functions, the test case should not care about the input and output of each function, instead, treat them as a whole functionality. In this way the implementation can be changed without affecting the test case, as long as it keeps the same interface.

Conclusion

In this post I explained “why TDD is bad”. Or in other words, the common pitfalls in using TDD and how to avoid them.

TDD is not necessarily a bad thing, but it is not silver bullet either. You have to find the right time and right place before using it. If you are in early product phases, and your product can be described with simple architecture, then TDD is a good choice to achieve fast iteration and quick response. However when you work on a mature product and the features are complicated or mission critical, think twice before practicing TDD. You need to carefully design the feature first, making some prototype to verify your design. Then you probably find TDD no longer necessary.

The concept of “test cases first” brought by TDD is really good. However before that don’t forget to put in some design efforts. And focus on behavior, not implementation.

Well that can hardly be called “Test-Driven” any more.

Thanks for your time reading this article.