|
Increasing the Effectiveness of Automated Testing
By: Shaun Smith and Gerard Meszaros
ABSTRACT
This paper describes techniques that can be used to
reduce the execution time and maintenance cost of the
automated regression test suites that are used to drive
development in eXtreme Programming (XP). They are
important because developers are more likely to write and
run test, thus getting valuable feedback, if testing is as
painless as possible. Test execution time can be reduced
by using an in-memory database to eliminate the latency
introduced by accessing a disk-based database and/or file
system. This paper also describes how the effort of test
development can be reduced through the use of a
framework that simplifies setup and teardown of text
fixtures.
1 INTRODUCTION
Automated testing is a useful practice in the toolkit of
every developer, whether they are doing XP or more
“traditional” forms of software development. The main
drawing card is the ability to rerun tests whenever you
want reassurance that things are working as required.
However, for tests to provide this valuable feedback to a
developer they have to run relatively quickly. On a series
of Java projects building business systems, we have
found that running large numbers of JUnit tests against a
relational database is normally too slow to provide the
kind of rapid feedback developers need. As a result we
have developed a collection of techniques, practices, and
technologies that together allow us to obtain the rapid
feedback developers need even when a rrelational
database underlies the system under construction.
2 DEVELOPER TESTING IS SUES
XP is heavily reliant on testing. Tests tell us whether we
have completed the implementation of some required
functionality and whether we have broken anything as a
result. On an XP project without extensive design
documentation, the tests, along with the code, become a
major means of communication. It is not just a case of
letting “the code speak to you”. The tests must speak to
you as well. And they should be listened to over and over
again.
Slow Test Execution
Having established that we rely so heavily on tests to
guide development and report on progress, it is no
surprise that we want to be able to run tests frequently for
feedback. We have found that it is essential for tests to
execute very quickly; our target is under 30 seconds for
the typical test run.
A purely economic argument is compelling enough by
itself. Assuming a developer runs the test suite every 10
minutes while developing, they will have run the suite 24
times in a single 4-hour programming session. A 1-
minute increase in test execution time increases
development time by 10% (24 minutes)! But the result of
slow tests is even more insidious than the economic
argument insinuates!
If test execution is too slow, developers are more likely to
put off testing and that delays the feedback. The
preferred model of development is the making of a series
of small changes, each time running the appropriate test
suite to assess the impact of the change. If tests run
slowly, developers will likely make a number of small
changes and then take a “test timeout” to run the tests.
The impact of this delayed use of tests is two fold. First,
debugging becomes more difficult because if tests fail
after a series of changes, identifying the guilty change is
difficult. Developers may even forget they made some
changes. Murphy’s Law says that this forgotten change
will most likely be the source of the failing tests.
Second, if tests take so long to run that a developer leaves
her desk while a test suite runs, her train of thought may
be broken. This routine of starting a test suite and then
getting up for a stretch and a chat was observed regularly
on one project. The lengthy test suite execution time was
because of the large number of tests and the relational
database access required by each test. This combination
became a recurring pattern on a number of projects.
Expensive Test Development
Given the test-first approach taken by XP, the cost of
writing and maintaining tests becomes a critical issue.
Unit tests can usually be kept fairly simple, but functional
tests often require large amounts of fixture setup to do
even limited testing. On several projects, we found that
the functional tests were taking large amounts of time to
write and needed considerable rework every time we
added new functionality. The complex tests were hard to
understand in part because they contained too much
necessary setup. We tried sharing previously setup
objects across tests, but we also found it difficult to write
tests that did not have unintended interactions with each
other.
Full article...
Other Resource
... to read more articles, visit http://sqa.fyicenter.com/art/
| 