Casey Muratori on Test Driven Development

Here are detailed notes from the transcript:

Detailed Notes: Critique of Test-Driven Development (TDD)

Main Topics Discussed:

• The effectiveness and methodology of software testing, particularly Test-Driven Development (TDD).
• The concept of "zero-sum" in programming time and resource allocation.
• The impact of development methodologies (like TDD and OOP) on software architecture and API design.
• Recommendations for a more effective approach to integrating testing into the development lifecycle.

Key Points and Arguments:

1. Distinction between Testing and Test-Driven Development (TDD):

   • Testing is good: The speaker explicitly states, "I think testing is very good" (0:41).
   • TDD is problematic: The core issue is "test driven development meaning forcing the programmer to think in terms of tests during development" (0:43-0:47). The "TD part" – the driving of development by tests – is the problem.
   • Analogy to OOP: The speaker likens TDD to Object-Oriented Programming (OOP). Having objects in code is fine, but "thinking in terms of [OOP]" wastes time and leads to "bad architecture" (0:24-0:37). Similarly, testing is fine, but driving development with tests is not.

2. Programming Time is Zero-Sum:

   • "Programming is zero sum" (0:17, 2:20, 3:28) is a central argument.
   • Time spent on one activity (e.g., making tests) is necessarily time not spent on another crucial activity (e.g., designing for use cases, optimizing performance, refining APIs).
   • This is presented as a fundamental tradeoff (2:01-2:10).

3. Negative Impacts of TDD:

   • Shifted Focus: TDD shifts the developer's primary focus away from actual use cases and problem-solving to thinking about tests (0:47-0:54, 3:11-3:14).
   • Poor API Design: If developers are spending time making interfaces revolve around tests, they are not spending time making them revolve around actual use cases (2:28-2:39). Actual use cases rarely look like tests.
   • Unusable Systems: This can lead to "a completely unusable mess" (2:40-2:46), specifically a messy or poorly designed API (2:55-2:57). The code itself might work and be well-tested, but its interface is difficult to use because it wasn't designed for its intended purpose (3:04).
   • Increased Complexity: Good APIs are already hard to make. Adding the TDD workflow "adds more complexity to the programmer's workflow," leading to a "worse result" (3:15-3:27).
   • Obsolescence: TDD can lead to "8,000 tests that no one ever needed because we ended up deleting that system anyway" (1:50-1:55), suggesting wasted effort on tests for features that don't persist.

4. Hypocrisy of TDD Advocates (The "TDD Paradox"):

   • People who advocate for TDD often "poo poo" (disparage) similar suggestions like "measuring performance during development" because it's seen as time-wasting (0:55-1:08).
   • The speaker argues this is inconsistent, as TDD itself consumes significant development time that could be spent elsewhere.

5. Real-World Observations and Lack of Efficacy:

   • Software from "TDD at this organization" often feels "full of bugs" (4:24-4:33), contrary to the promised benefit of TDD.
   • The speaker notes that despite extensive testing and code reviews (presumably), widespread issues persist in high-profile software.

Important Facts or Data Mentioned:

• "Programming is zero sum." (Repeated principle)
• "8,000 tests that no one ever needed because we ended up deleting that system anyway?" (Anecdotal example of wasted testing effort)
• YouTube Play Button Bug: For "the past eight years, the play button cannot properly represent whether something is playing or not" (4:41-4:47). This specific example is used to illustrate that extensive testing (likely done at YouTube) doesn't guarantee a bug-free or intuitive user experience.

Conclusions or Recommendations:

1. Rethink Time Allocation: Developers and organizations need to reconsider how they allocate programming time, recognizing its zero-sum nature (5:01-5:06).
2. Test Later and Strategically: Testing should occur after the core system takes shape and its design is stable:
   • Once the system works properly, the API is satisfactory, and performance targets are reasonably met (3:29-3:48).
   • Tests should then be added strategically to specific parts of the system (3:48-3:54).
3. Focus Tests on Critical Areas: Prioritize testing for:
   • Parts "whose bugs will be either very hard to find" (4:00-4:03).
   • Parts with "really catastrophic" bugs that could lead to significant financial loss or user system crashes (4:03-4:06).
4. Avoid Test-Driven Development: TDD is considered "a really bad idea" (4:20-4:22) because it misdirects developer focus and resource allocation.