Software QA FYI - SQAFYI

Retrofitting an Acceptance Test Framework for Clarity

By: Rick Mugridge and Ewan Tempero

Abstract
An XP customer needs to write and check acceptance tests. However, the format for defining the tests needs to be clear. Many acceptance test approaches use arcane formats which do not promote clarity for the customer, due to a conflict of interest between the complexities of automation and the needs of the customer. We discuss the evolution of acceptance tests to improve their clarity for the customer.

Sat is an acceptance test system for testing socketbased servers with multiple clients. The first version used an XML file to define the tests in a test suite. Any errors detected were written to a text log. There were two problems with this first version. The XML format made it difficult to read and edit the tests. When an error was given, it was not easy to identify the place in the test where the problem occurred.
Sat was altered to make use of Fit, a testing framework that uses HTML tables for defining tests and reporting any errors. We found the new version considerably easier to use. The tabular form makes it much simpler to read and alter the tests. Any errors are reported in a copy of the tables, in the place where they occur. We have also found it convenient to include information about the tests in the HTML, providing a form of "literate testing".

1. Introduction
The practices of XP bring testing to focus early in software development, rather than catching problems well after they have occurred [4]. Tests play a significant role in development, with acceptance (or customer) tests helping to drive story development and with programmer tests driving design. Much of what has been learned about practical testing in traditional software development approaches [2] also apply in XP, with testers providing a cogent perspective [12].

In XP, acceptance tests are ideally written by the customer with help from testers. Such test may need to evolve as the needs of the system are better understood Hence it is essential to use a format that is both suitable for automatic execution and which a customer can easily understand. A syntactical format that is well suited to automatic execution may take a lot of effort to learn, more than many customers can afford. Descriptions of the tests also need to be easy to read and understand, so that they can be verified as being correct and complete [2]. In this paper, we discuss the development of clarity in acceptance tests, and in particular explore the use of the Fit system for specifying them [5].

Sat (Socket Acceptance Tester) is an acceptance (customer) test system for testing socket-based servers with multiple clients. We initially developed Sat to define customer tests for a Chat server that was developed in a student XP project in 2002.

The first version of Sat used an XML file to define the tests in a test suite, where each test consisted of a sequence of messages being sent and expected to be received by one or more clients. Any errors detected, such as when an unexpected message was received by a client, were written to a text log.

We found several problems with the first version. The XML format made it difficult to read (and edit) the message sequence in a test and to keep track of the expected state of the server after each message was sent to it. When an error was given, it wasn't easy to identify the place in the test where the problem occurred. At this time, we became aware of the Fit framework [5]. It was immediately obvious that using the table format of Fit for both defining the tests and for reporting would be superior to the XML and the logs that we had been using. The Fit table format would also allow a customer to write the tests with a simple HTML editor.

We altered Sat to make use of Fit. The customer found the new version much easier to use. The tabular form of the tests, with a table column for each client, make it much easier to read, create and edit customer tests. Any errors are signalled in a copy of the HTML, in the tables where they occur. This makes it much easier to understand what test has failed and with what symptoms. It is convenient to add information about the tests outside the tables, providing a form of "literate testing" akin to literate programming [9].

We follow with an introduction to the Chat server. Section 3 introduces the first version of Sat, shows example tests for the chat server, and discusses its limitations. Section 4 introduces the Fit framework. Section 5 discusses the second version of Sat, along with revised tests. Section 6 discusses related work and Section 7 concludes.

Full article...


Other Resource

... to read more articles, visit http://sqa.fyicenter.com/art/

Retrofitting an Acceptance Test Framework for Clarity