Software QA FYI - SQAFYI

Increasing the Effectiveness of Automated Testing

By: Shaun Smith and Gerard Meszaros

ABSTRACT This paper describes techniques that can be used to reduce the execution time and maintenance cost of the automated regression test suites that are used to drive development in eXtreme Programming (XP). They are important because developers are more likely to write and run test, thus getting valuable feedback, if testing is as painless as possible. Test execution time can be reduced by using an in-memory database to eliminate the latency introduced by accessing a disk-based database and/or file system. This paper also describes how the effort of test development can be reduced through the use of a framework that simplifies setup and teardown of text fixtures.

Automated testing is a useful practice in the toolkit of every developer, whether they are doing XP or more “traditional” forms of software development. The main drawing card is the ability to rerun tests whenever you want reassurance that things are working as required. However, for tests to provide this valuable feedback to a developer they have to run relatively quickly. On a series of Java projects building business systems, we have found that running large numbers of JUnit tests against a relational database is normally too slow to provide the kind of rapid feedback developers need. As a result we have developed a collection of techniques, practices, and technologies that together allow us to obtain the rapid feedback developers need even when a rrelational database underlies the system under construction.

XP is heavily reliant on testing. Tests tell us whether we have completed the implementation of some required functionality and whether we have broken anything as a result. On an XP project without extensive design documentation, the tests, along with the code, become a major means of communication. It is not just a case of letting “the code speak to you”. The tests must speak to you as well. And they should be listened to over and over again.

Slow Test Execution
Having established that we rely so heavily on tests to guide development and report on progress, it is no surprise that we want to be able to run tests frequently for feedback. We have found that it is essential for tests to execute very quickly; our target is under 30 seconds for the typical test run.

A purely economic argument is compelling enough by itself. Assuming a developer runs the test suite every 10 minutes while developing, they will have run the suite 24 times in a single 4-hour programming session. A 1- minute increase in test execution time increases development time by 10% (24 minutes)! But the result of slow tests is even more insidious than the economic argument insinuates!

If test execution is too slow, developers are more likely to put off testing and that delays the feedback. The preferred model of development is the making of a series of small changes, each time running the appropriate test suite to assess the impact of the change. If tests run slowly, developers will likely make a number of small changes and then take a “test timeout” to run the tests. The impact of this delayed use of tests is two fold. First, debugging becomes more difficult because if tests fail after a series of changes, identifying the guilty change is difficult. Developers may even forget they made some changes. Murphy’s Law says that this forgotten change will most likely be the source of the failing tests. Second, if tests take so long to run that a developer leaves her desk while a test suite runs, her train of thought may be broken. This routine of starting a test suite and then getting up for a stretch and a chat was observed regularly on one project. The lengthy test suite execution time was because of the large number of tests and the relational database access required by each test. This combination became a recurring pattern on a number of projects. Expensive Test Development Given the test-first approach taken by XP, the cost of writing and maintaining tests becomes a critical issue. Unit tests can usually be kept fairly simple, but functional tests often require large amounts of fixture setup to do even limited testing. On several projects, we found that the functional tests were taking large amounts of time to write and needed considerable rework every time we added new functionality. The complex tests were hard to understand in part because they contained too much necessary setup. We tried sharing previously setup objects across tests, but we also found it difficult to write tests that did not have unintended interactions with each other.

Full article...

Other Resource

... to read more articles, visit

Increasing the Effectiveness of Automated Testing