Software QA FYI - SQAFYI

Diagnosing evolution in test-infected code

By: Christian Wege,Martin Lippert

In this study we trace the effects of applying the techniques of refactoring and aggressive unit testing in source code based on historical information. We show how their impact on the evolution of the architecture can be testified. The study comprises the analysis of a large number of indiv idual integration versions of a large framework. The method described here can help development teams find weaknesses in their application of the two traced techniques.

In a world of constantly changing requirements, systems development must ensure that it is able to quickly respond to changed user requirements or technology updates. One major promise of Extreme
Programming(XP) [2] is to enable the construction of an evolvable system. Instead of planning all possible future enhancements from the very beginning an extreme programmer relies on its ability to incorporate changes to the system and its architecture at an arbitrary point in the future.
In this study we investigated the artifacts (namely the source code and other historical information) of a project which uses the aggressive unit testing and refactoring techniques extensively for the development. We trace the effects of the application of these two techniques in the developed source code. Our system under investigation is JWAM1 – a framework for constructing large scale interactive software systems.
In their well-known article Beck and Gamma introduce the testing style of test infection [1]. For every class in the system you write a unit test. New requirements are implemented in the system by refactoring the unit tests first and then the system classes [4]. So when those two techniques are applied strictly we talk about test infected code. Given this definition JWAM is test-infected. Its test

suite created with the Java testing framework provided by Beck and Gamma2.
The JWAM development relies on an integration server [7] which ensures that for every update of the source code all tests still run. The study is based on 254 individual integration versions of the framework which stem from this continuous integration process. In addition to the source code we used the integration log which contains a small description for every update of the source tree. Lippert et al. state that “With Pair Programming we have improved framework quality, with test cases we maintain it. Without the test cases a lot of the refactoring we did in the past would have been less smooth.”[8] With the help of our analysis we validated this rather intuitive statement by observing the history of specific system properties in the produced artifacts. As well with the help of our analysis we can point out some areas of potential improvements of the framework development.

This study concentrates on tracing the effects of agressive unit testing and refactoring directly in the code and in historical information. It doesn’t investigate the correlation to requirements changes of defect rates, which would be of high interest as well.

The method
The method used in our study is an adaptation of the approach proposed by Mattsson and Bosch [9] for observing software evolution in object-oriented frameworks. Based on historical information about the subsystems, modules and classes they investigated the size, change rate and growth rate of the system. The work of Mattsson and Bosch is based on a method proposed in [5] which focus was on observing the macro-level software evolution using the version numbering of a system. Mattsson and Bosch adapted this approach for investigating object-oriented frameworks. The system was divided into a number of subsystems which were themselves divided in several modules. In the adapted approach each module consisted of several classes (instead of programs as in the original approach). Size is calculated by the number of classes in each module or subsystem. The calculations of change and growth rate are made in terms of changed classes as units. Class change is measured in terms of the change in the number of public methods for each class. The focus on public methods stems from the fact that a change in the public methods reflects a better understanding of the boundary of the system. Changes of private methods however ma inly reflect refinements of implementation details and are thus of minor interest.

The method steps in the original approach are3:
1. calculate, for all releases, the change and growth rate for the whole system,
2. calculate, for all releases, the change and growth rate for each of the subsystems,
3. for those subsystems that exhibit high growth and change rates calculate, for all releases, the change and growth rates for the modules,
4. those modules that exhibit high change and growth rates are identified as likely candidates for restructuring [9].

For our study we based our calculations for system, subsystems and modules on the package/subpackage structure of Java. The packaging feature of Java is a natural structuring mechanism provided by the language. In JWAM this mechanism is used to distinguish between the core and several non-core part of the whole system and inside the core to distinguish between the framework layers. We will discuss this in more detail later. Java interfaces are treated exactly the same way as Java classes.

The second important adaptation is that we changed the top-down approach to a bottom-up approach. Instead of starting with the top level system, we calculate the values for every class and subsystem and go up to the top. We try to trace the development method in the code, therefor we are interested in all developed artifacts. For being able to give advice on possible restructuring candidates (like in the approach by Mattsson and Bosch) we have to widen the empirical base first.

The third and most important adaptation is the introduction of the test coverage rate. If aggressive unit testing is one central part of test-infected programming then the results should be dependent on the number of system classes covered by unit tests.

The investigated system
JWAM is a Java framework supporting the development of large scale interactive software systems according to the tools & materials approach [11]. The foundation of the JWAM framework was laid in 1997 by research assistants and students of the Software Engineering Group at the University of Hamburg [8]. In 1998 the commercialization of the framework began. In 1999 the

Full article...

Other Resource

... to read more articles, visit

Diagnosing evolution in test-infected code