Software QA FYI - SQAFYI

Agile Regression Testing Using Record & Playback

By: Gerard Meszaros, Ralph Bohnet, Jennitta Andrea

Abstract. There are times when it is not practical to hand-script automated tests for an existing system before one starts to modify it. In these circumstances, the use of “record & playback” testing may be a viable alternative to hand-writing all the tests. This paper describes experiences using this approach and summarizes key learnings applicable to other projects.

1 Introduction
Scripting tests by hand as required by JUnit or it’s siblings is hard work and requires special skills to write tests. Writing functional (or acceptance) tests using JUnit is particularly hard because of all the data requirements. One possible alternative is the FIT frameworks [1] but these still require someone to develop utility code to act as “glue” between the testing framework and the system under test (SUT). Each of these approaches requires that the system provide an interface by which the tests can be conducted.

1.1 Catch-22 of XUnit-Based Testing
On several recent agile projects, we found ourselves needing to modify an existing system that had no automated tests. In one case, the manual retest effort was expected to involve several person-years of effort and many months of elapsed time. Handscripting JUnit (or equivalent) tests was considered too difficult because the systems were not designed for testability. (e.g. business logic embedded in the UI; no access to an application API; no means of controlling all the test setup (such as “stubbing”)). Refactoring the system for testability so that tests could be written was considered too risky without having automated regression testing to verify that the refactoring had not introduced problems. And even if we had done it, we were concerned that we could not hand-script all the necessary tests, complete with their expected outcomes, in the time and resource budget available to us.

1.2 Looking for Alternatives to XUnit
This led us to investigate all possible alternatives to XUnit style testing. Most of this effort focused on “record & playback” (R&PB) styles of test creation, an approach that involved recording functional tests on the system before we made the changes and regression testing the refactored system by playing back these tests. The tests verified the overall functionality of the system and, in particular, much of the business logic it contained. Once the tests were recorded and verified (successfully played back on the original system), we could then start refactoring the system to improve its design. We felt that this would allow us to quickly record a number of tests that we could play back at will. Since we had an existing version of the system to use as a “gold standard”, we could leave the effort of defining the expected outcomes to the record and playback framework.
This paper describes the options we considered, their advantages and disadvantages and how we ended up regression testing the system. The work also lead to an understanding of where R&PB can be used in a more general context than the specific projects with which we were dealing.
2 Issues with R&PB Test Automation
R&PB style testing predates XUnit-style testing by many decades. Test automation folklore is rich with horror stories of failed attempts to automate testing. This paper describes critical success factors for making this style of testing work, what to avoid, and best practices in R&PB test automation.

The “robot user” approach to test automation had received enough bad publicity in past attempts at test automation that we found it to be a hard “sell”. We had to convince our sponsors that “this time it would be different” because we understood the limitations of the approach and that we had a way to avoid the pitfalls.

2.1 The “Fragile Test” Problem
Test automation using commercial R&PB or “robot user” tools has a bad reputation amongst early users of these tools. Tests automated using this approach often fail for seemingly trivial reasons. It is important to understand the limitations of this approach to testing to avoid falling victim to the common pitfalls. These include Behavior Sensitivity, Interface Sensitivity, Data Sensitivity and Context Sensitivity.
Behavior Sensitivity
If the behavior of the system is changed (e.g. the requirements are changed and the system is modified to meet the new requirements), any tests that exercise the modified functionality will most likely fail when replayed. This is a basic reality of testing regardless of the test automation approach used.
Interface Sensitivity
Commercial R&PB (“robot user”) test tools typically interact with the system via the user interface. Even minor changes to the interface can cause tests to fail even though a human user would say the test should still pass. This is partly what gave test automation tools a bad name in the past decade.

Data Sensitivity
All tests assume some starting point; these are often called the “pre-conditions” or “before picture” of the test. Most commonly, this is defined in terms of data that is already in the system. If the data changes, the tests may fail unless great effort has been expended to make the tests insensitive to the data being used. More recent versions of the test automation tools provide mechanisms that can be used to make tests less sensitive. This has added a lot of complexity to these tools and, as a result, they often fail to live up to their promises. This has likely contributed to the bad reputation they have received.

Context Sensitivity
The behavior of the system may be affected by the state of things outside the system. This could include the states of devices (e.g. printers, servers) other applications, or even the system clock. E.g. the time and/or date of test.

Full article...

Other Resource

... to read more articles, visit

Agile Regression Testing Using Record & Playback