Automated Exploratory Testing

By: Ayal Zylberman and Nitzan Shenar

Automated exploratory testing is a method integrating Test Automation within the Exploratory Testing Session that enables the testers to save money and to become more productive during test execution. This article describes automated exploratory testing and explains its benefits.

What is Exploratory Testing?
The most common definition of exploratory testing was provided by James Bach: Exploratory testing is simultaneous learning, test design, and test execution (Exploratory Testing Explained by James Bach, v.1.3)

Exploratory testing is distinguished from Scripted testing, which represent the traditional way of performing tests in which the test process is divided into several activities such as Study, Test Documentation and Test Execution that have to be performed in a sequential way (for example: Test Execution can not start until all test documentations are ready).

Many people tend to get confused between exploratory testing and Ad-hoc testing. Ad-hoc testing is a way to perform testing without any documentation at all. Usually, when the main reason why organizations perform Ad-hoc testing is because time and budget constraints prevent them from complying with the requirements of the scripted testing process, so shortcuts need to be implemented. Completely unplanned random actions on an application is not exploratory testing but rather, bad testing (Exploratory Testing by Peter Marshall).

Session Based Testing
Test Session is the most common approach of implementing exploratory testing. In session based testing, testing occurs in special time-boxes called test sessions. A test session is a period of uninterrupted time where exploration occurs, usually 60-120 minutes long. Each test session contains a specific portion of what needs to be tested. A test session can be a business process of the AUT, a screen, a DB table or any other components that need to be addresses as part of testing.

Pair Exploratory Testing
A common approach of implementing a pair exploratory testing is Pair Testing when exploratory testing is done with two testers and (typically) one machine. The most common implementation of Pair testing is when one tester is sitting at the keyboard and explaining the test ideas and hypothesis which emerge, while another sits alongside, taking notes and suggesting additional ideas along the way.

Main problems and risks with Exploratory Testing
As a testing service provider I found it hard to convince many of my clients to use exploratory testing. I was facing a lot of resistance to the concept of exploratory testing. Some of the arguments were mainly about the inherent resistance for change at least for conservative people that are used to perform testing in a certain way for many years. However, I found some of the arguments reasonable and thus, I addressed those arguments when planning the test strategy proposed to those organizations. Although I’m not sure the risks below should be considered when underlying an appropriate test strategy to an organization, I included them in the article since organizational and political constraints needs to be addressed as part of the testing activity (I once defined organization politics as any activity performed by an organization that does not comply with its interests):
* Coverage Review – deep auditing and review of the test coverage (at least prior to test execution) is missing in exploratory testing and hence increase the chance that parts of the AUT will not be tested.
* Visibility and transparency – many organizations use the test documentation as a way to achieve visibility and transparency prior to test execution. Metrics (such as amount of test cases written in each day) are used in order to monitor the performance and progress of the team.
* Resources and Time Constraints during test execution – usually organization tend not to allocate enough time for the test execution. Even when appropriate time is allocated, delays in development cause the test execution period to shrink, sometimes by more than 50%. Since exploratory testing requires more time and resources than scripted testing, this might cause low coverage of testing.

Automating the exploratory testing, as described later on in this article, will help organizations to mitigate the above risks.

What is Test Automation?
Test automation is a process in which a tool is being used in order to support a certain test activity. The most common test automation tools are GUI Test Automation such as QTP, TestPartner, Robot, TestComplete and many more. Those tools are mainly used in order to automate the test execution activity. Using those tools the organization can create a test script that performs actions on the GUI of the AUT, so later on those scripts can be executed in each one of the test execution iterations instead of the testers.

Keyword Driven Testing (KDT)
Keyword Driven Testing is a comprehensive, cross-organization test design approach that bridges the gap between the Test Automation team and the rest of the roles involved in the testing process. It allows both manual testers and other subject-matter experts to design, build and execute test automation scripts without any programming knowledge. KDT approach divides the test automation infrastructure layer from the logical layer. The infrastructure layer contains set of scripts that are triggered by a call made from the logical layer (i.e. Keyword). The logical layer, developed by the manual testers, is a set of keywords that formalize an automated test script.

Full article...

Other Resource

... to read more articles, visit http://sqa.fyicenter.com/art/