Retrofitting an Acceptance Test Framework for Clarity
By: Rick Mugridge and Ewan Tempero
An XP customer needs to write and check acceptance
tests. However, the format for defining the tests needs to
be clear. Many acceptance test approaches use arcane
formats which do not promote clarity for the customer,
due to a conflict of interest between the complexities of
automation and the needs of the customer. We discuss
the evolution of acceptance tests to improve their clarity
for the customer.
Sat is an acceptance test system for testing socketbased
servers with multiple clients. The first version used
an XML file to define the tests in a test suite. Any errors
detected were written to a text log. There were two
problems with this first version. The XML format made it
difficult to read and edit the tests. When an error was
given, it was not easy to identify the place in the test
where the problem occurred.
Sat was altered to make use of Fit, a testing
framework that uses HTML tables for defining tests and
reporting any errors. We found the new version
considerably easier to use. The tabular form makes it
much simpler to read and alter the tests. Any errors are
reported in a copy of the tables, in the place where they
occur. We have also found it convenient to include
information about the tests in the HTML, providing a
form of "literate testing".
The practices of XP bring testing to focus early in
software development, rather than catching problems well
after they have occurred . Tests play a significant role
in development, with acceptance (or customer) tests
helping to drive story development and with programmer
tests driving design. Much of what has been learned
about practical testing in traditional software development
approaches  also apply in XP, with testers providing a
cogent perspective .
In XP, acceptance tests are ideally written by the
customer with help from testers. Such test may need to
evolve as the needs of the system are better understood
Hence it is essential to use a format that is both
suitable for automatic execution and which a customer can
easily understand. A syntactical format that is well suited
to automatic execution may take a lot of effort to learn,
more than many customers can afford. Descriptions of the
tests also need to be easy to read and understand, so that
they can be verified as being correct and complete . In
this paper, we discuss the development of clarity in
acceptance tests, and in particular explore the use of the
Fit system for specifying them .
Sat (Socket Acceptance Tester) is an acceptance
(customer) test system for testing socket-based servers
with multiple clients. We initially developed Sat to define
customer tests for a Chat server that was developed in a
student XP project in 2002.
The first version of Sat used an XML file to define the
tests in a test suite, where each test consisted of a
sequence of messages being sent and expected to be
received by one or more clients. Any errors detected,
such as when an unexpected message was received by a
client, were written to a text log.
We found several problems with the first version. The
XML format made it difficult to read (and edit) the
message sequence in a test and to keep track of the
expected state of the server after each message was sent
to it. When an error was given, it wasn't easy to identify
the place in the test where the problem occurred.
At this time, we became aware of the Fit framework .
It was immediately obvious that using the table format of
Fit for both defining the tests and for reporting would be
superior to the XML and the logs that we had been using.
The Fit table format would also allow a customer to write
the tests with a simple HTML editor.
We altered Sat to make use of Fit. The customer found
the new version much easier to use. The tabular form of
the tests, with a table column for each client, make it much
easier to read, create and edit customer tests. Any errors
are signalled in a copy of the HTML, in the tables where
they occur. This makes it much easier to understand what
test has failed and with what symptoms. It is convenient
to add information about the tests outside the tables,
providing a form of "literate testing" akin to literate
We follow with an introduction to the Chat server.
Section 3 introduces the first version of Sat, shows
example tests for the chat server, and discusses its
limitations. Section 4 introduces the Fit framework.
Section 5 discusses the second version of Sat, along with
revised tests. Section 6 discusses related work and
Section 7 concludes.
... to read more articles, visit http://sqa.fyicenter.com/art/