• Realizing quality improvement through test driven development ...

Empir Software Eng (2008) 13:289–302
DOI 10.1007/s10664-008-9062-z
Realizing quality improvement through test driven
development: results and experiences of four industrial
teams
Nachiappan Nagappan & E. Michael Maximilien &
Thirumalesh Bhat & Laurie Williams
Published online: 27 February 2008
# Springer Science + Business Media, LLC 2008
Editor: Pankaj Jalote
Abstract Test-driven development (TDD) is a software development practice that has been
used sporadically for decades. With this practice, a software engineer cycles minute-by-minute
between writing failing unit tests and writing implementation code to pass those tests. Test-
driven development has recently re-emerged as a critical enabling practice of agile software
development methodologies. However, little empirical evidence supports or refutes the utility
of this practice in an industrial context. Case studies were conducted with three development
teams at Microsoft and one at IBM that have adopted TDD. The results of the case studies
indicate that the pre-release defect density of the four products decreased between 40% and
90% relative to similar projects that did not use the TDD practice. Subjectively, the teams
experienced a 15–35% increase in initial development time after adopting TDD.
Keywords Test driven development . Empirical study . Defects/faults . Development time
1 Introduction
Test-driven development (TDD) (Beck 2003) is an “opportunistic” (Curtis 1989) software
development practice that has been used sporadically for decades (Larman and Basili 2003;
Editor: Pankaj Jalote
N. Nagappan (*) : T. Bhat
Microsoft Research, Redmond, WA, USA
e-mail: nachin@microsoft.com
T. Bhat
e-mail: thirub@microsoft.com
E. M. Maximilien
IBM Almaden Research Center, San Jose, CA, USA
e-mail: maxim@us.ibm.com
L. Williams
North Carolina State University, Raleigh, NC, USA
e-mail: williams@csc.ncsu.edu
290 Empir Software Eng (2008) 13:289–302
Gelperin and Hetzel 1987). With this practice, a software engineer cycles minute-by-minute
between writing failing unit tests and writing implementation code to pass those tests. Test-
driven development has recently re-emerged as a critical enabling practice of agile software
development methodologies (Cockburn 2001), in particular Extreme Programming (XP;
Beck 2005). However, little empirical evidence supports or refutes the utility of this practice
in an industrial context.
In this paper we report on “in vivo” case studies of four software development teams that
have adopted the TDD or TDD-inspired practice, one team from IBM (Williams et al. 2003)
and three from Microsoft (Bhat and Nagappan 2006). We study TDD as a stand alone
software development practice; in that, we do investigate TDD within the prevailing
development processes at IBM and Microsoft and not within the context of XP. In this
paper, we share details about the projects, the teams, and how they used TDD in their
projects. We compare the quality and development time of the product developed by a team
using TDD relative to a similar team that did not use TDD. Finally, we provide lessons
learned by these four teams.
Case studies can be viewed as “research in the typical,” (Kitchenham et al. 1995;
Fenton and Pfleeger 1998) increasing the degree to which the results can be compared
with those of other industrial teams. However, case studies cannot be performed with the
rigor of experiments because they involve real people, real projects, and real customers
over a relatively long period of time. In empirical research, replication of studies is a
crucial factor to building an empirical body of knowledge about a practice or a process.
The four studies presented in this paper contribute towards strengthening of the empirical
knowledge of the efficacy of TDD because each case study was conducted in a different
context, as shown Fig. 1. The details of the contexts of each case study are discussed
throughout the paper.
The rest of this paper is organized as follows. Section 2 gives an overview of TDD and
Section 3 the related works. Section 4 explains the context variables of our projects, and
Section 5 their individual TDD implementation practices. Section 6 presents the results of
the study and Section 7 the threats to validity. Section 8 presents the lessons learned for the
benefits of organizations new to TDD or thinking about adopting the practice.
Fig. 1 Varying context factors of the four case studies
Empir Software Eng (2008) 13:289–302 291
2 Test Driven Development
When discussing TDD we consider a task to be a subset of a requirement that can be
implemented in a few days or less (Beck and Fowler 2001). As illustrated in Fig. 2, TDD
software engineers develop production code through rapid iterations of the following:
Minute-by-minute cycles:
– Writing a small number of new and failing automated unit test case(s) for the task at
hand
– Implementing code which should allow the new unit tests cases to pass
– Re-running the new unit test cases to ensure they now pass with the new code
Per-task cycles:
– Integrate code and tests for new code into existing code base
– Re-running all the test cases in the code base to ensure the new code does not break
any previously running test cases
– Refactoring the implementation or test code (as necessary)
– Re-running all tests in the code base to ensure that the refactored code does not break
any previously passing test cases
In TDD, code development is kept within the developer’s intellectual control because he or she
is continuously making small design and implementation decisions and increasing functionality
at a relatively consistent rate. New functionality is not considered properly implemented unless
the new unit test cases and every other unit test cases written for the code base run properly.
Some possible benefits of TDD are:
– Better Design. TDD is considered as much of (and in some cases, more of) a design
process than a test process. Kent Beck, the person responsible for the inclusion of TDD
in XP and Ward Cunningham, an early proponent of TDD in an XP context,
immediately stated, “Test-first coding is not a testing technique” (Beck 2001; Note: the
test-first coding practice eventually came to be known as TDD). They assert that TDD
Fig. 2 TDD methodology overview
292 Empir Software Eng (2008) 13:289–302
is an analysis and design technique and that: “Test-first code tends to be more cohesive
and less coupled than code in which testing isn’t part of the intimate coding cycle”
(Beck 2001).
– Efficiency. The fine granularity of test-then-code cycle gives continuous feedback to
the developer. With TDD, faults and/or defects are identified very early and quickly as
new code is added to the system. Consequently, the source of the problem is also more
easily determined since the problems have not been in existence for long. We contend
that the efficiency of fault and defect removal and the corresponding reduction in the
debugging and maintenance time compensates for the additional time spent writing and
executing test cases.
– Test Assets. The automated unit test cases written with TDD are valuable assets to the
project. Subsequently, when the code is enhanced, modified, or updated, running the
automated unit tests may be used for the identification of newly introduced defects, i.e.,
for regression testing.
– Reducing Defect Injection. Unfortunately, maintenance fixes and “small” code changes
may be nearly 40 times more error prone than new development (Humphrey 1989), and
often, new faults are injected during the debugging and maintenance phases. The ease
of running the automated test cases after changes are made should also enable smooth
integration of new functionality into the code base and therefore reduce the likelihood
that fixes and maintenance changes introduce new defects. The TDD test cases are
essentially a high-granularity, low-level, regression test. By continuously running these
automated test cases, one can find out whether a change breaks the existing system
early on, rather than leading to a late discovery.
3 Related Works
We briefly summarize the relevant empirical studies on TDD and their high level results.
George and Williams (George and Williams 2003a, b) performed a structured experiment
involving 24 professional programmers from John Deere, Rolemodel Software, and
Ericsson, to investigate the efficacy of TDD. One group developed a small Java program
using TDD while the other control group used a waterfall-like approach. Experimental
results indicated that TDD programmers produce higher quality code because they passed
18% more functional black-box test cases. On the other hand, the TDD programmers took
16% more time. However, the programmers in the control group often did not write the
required automated test cases after completing their code.
Müller and Hagner (2002) found that a TDD-variant did not help produce a higher
quality system. In this study, the programmers (graduate computer science students) had to
write their complete set of automated unit test cases before writing any production code
rather than in the highly iterative manner of TDD. Erdogmus et al. (2005) performed a
controlled investigation regarding test-first and test-last programming using 24 undergrad-
uate computer science students. They observed that TDD improved programmer
productivity but did not, on average, help the engineers to achieve a higher quality
product. Their conclusions also brought out a valid point that the effectiveness of the test-
first technique depends on the ability to encourage programmers to enhance their code with
test assets. Müller and Tichy (2001) investigated XP in a university context using 11
students. From a testing perspective they observed that, in the final review of the course,
87% of the students stated that the execution of the test cases strengthened their confidence
in their code.
Empir Software Eng (2008) 13:289–302 293
Janzen and Seiedian (2006) conducted an experiment with undergraduate students in a
software engineering course. Students in three groups completed semester-long program-
ming projects using either an iterative test-first (TDD), iterative test-last, or linear test-last
approach. Results from this study indicate that TDD can be an effective software design
approach improving both code-centric aspects such as object decomposition, test coverage,
and external quality, as well as developer-centric aspects, which includes productivity and
confidence.
Our work builds up on the prior empirical work in this area. Few case studies examine
the use of TDD in an industrial context. Our paper contributes to the empirical base of
knowledge about the TDD practice by combining results of four empirical studies carried
out in different industrial contexts and environments.
4 The Projects and the Teams
In this section, we provide information about the four products and teams and the contexts
into which the projects were executed. As a cautionary note, as with all empirical research,
the results should not be taken to be valid in the general sense, only in the context in which
the case study was conducted.
4.1 IBM
The IBM case study was conducted with a group that has been developing device drivers
for over a decade. They have one legacy product that has undergone seven releases since
late 1998. This legacy product was used as the baseline in our case study. In 2002, the
group developed the device drivers on a new platform and moved to a completely new
architecture. In our case study, we compare the seventh release of the legacy platform with
the first release of the new platform. Because of its longevity, the legacy system handles
more classes of devices, on multiple bus connectivity, with more vendors’ devices than the
new system. Hence, while not a true control group, the legacy software can still provide a
valuable relative insight into the performance of the TDD methodology since the two
products exposed the same interfaces for the same set of devices—a JavaPOS1 interface.
The device drivers from the legacy code base were wrapped using JNI to expose the Java™
interface, and in the new system, the device drivers were mostly entirely implemented in
the Java technology.
4.2 Microsoft
The three case studies at Microsoft were distributed across three of the main product
families, Windows, MSN, and Visual Studio (Bhat and Nagappan 2006). The TDD and
non-TDD projects are developed by teams led by project managers with similar levels of
responsibilities and reporting to the same higher-level manager. As a result, our
comparisons involve projects in a similar usage domain. We therefore avoid making
unequal comparisons, as would be the case in comparing, for example, computer game and
nuclear reactor software. Also, these projects were analyzed post hoc, and the developers
did not know during development that their work was going to be assessed. Therefore, the
case study did not interfere with the developers’ performance. There was no enforcement
1
http://www.javapos.org
294 Empir Software Eng (2008) 13:289–302
and monitoring of the TDD practice; and decisions to use TDD were made as a group.
More detail is provided on the projects completed by each of the three teams:
& Windows: This case study was performed in the Networking team. The networking
common library was written by the tool development team as a re-usable set of
modules for different projects within the networking tools team. Some examples
are generic packet manipulation routines, command line manipulation, memory
allocation and tracking routines, random number generator, and timer classes. This
library is used by more than 50 internal applications.
& MSN: This case study involves a Web services application in the MSN division.
MSN is a business division of Microsoft that works on Web service applications
that range from online shopping to news and Web search technologies.
& Developer Division: This project is part of the Visual Studio (VS) software
development system. Visual Studio provides a range of tools for individual
developers and software development teams. Visual Studio allows for the
development of stand-alone applications, Web service applications, and so on.
4.3 Examining the Range of Teams and Products
Tables 1 and 2 respectively show a summary of a comparison between the team and the product
measures. The team factors enable us to understand the composition of these four teams. The
product factors present a comparative view of the size and effort associated with the projects.
Across the four projects, we observe that the teams are experienced; the majority of the
team members had around 6–10 years experience. In the MSN team, the developers are
experienced but only have moderate domain and programming language expertise. The
IBM project has four developers who were inexperienced. Further, the IBM project was
developed in a distributed context with five developers in Raleigh, NC and four in
Guadalajara, Mexico. No one on the new IBM team knew TDD beforehand, and three were
somewhat unfamiliar with Java. In the IBM case, all but two of the nine full-time
developers were novices to the targeted devices.
Table 1 Summary of team factors
Metric description IBM: Drivers Microsoft: Microsoft: Microsoft:
Windows MSN VS
Team size 9 6 5–8 7
Team location USA and Mexico Redmond, Redmond, Redmond,
WA WA WA
Experience level >10 years 0 0 1 5
6–10 years 3 5 7 0