Software QA FYI - SQAFYI

An Introduction to Software Test Automation

By: Saket Godase

Software Testing
Though there are many formal definitions of software testing it basically boils down to “Software testing is a process used to identify the correctness, completeness and quality of developed computer software.” In reality, testing can never establish the correctness of the software. It can only find defects, not prove that there are none. The testing process is carried out either manually or using automation tools or an effective combination of both.

Manual Testing – is defined as developing and executing tests that rely primarily on direct human interaction throughout the entire test case, especially in terms of evaluating correctness and ascertaining test status.

Automation Testing – is defined as developing and executing tests that can run unattended, comparing the actual to expected results and logging status. The rest of the paper will concentrate primarily on automation testing.

Simply put, "Automated Testing" is automating the manual testing process currently in use. Minimal setup includes:
* Detailed Test Cases
* Predictable “Expected Results”
* Dedicated Test Environment
* And most importantly dedicated and skilled resources.

Automated testing is expensive in terms of cost. It does not replace the need for manual testing or enable you to "down-size" your testing department. Automated testing is an addition to your testing process.

However, test automation can be cost-effective by following some of the time-tested techniques given below.

* Choose a test tool that best fits the testing requirements of your organisation. A good place to start is "Automated Testing Handbook" available from the Software Testing Institute, which covers all of the major considerations involved in choosing the right test tool for your purposes.
* One-time tests are not worth automating. Only automate those tests that are going to be repeated.
* Before starting off with Test Automation, identify the test cases that can be automated.
* Concentrate on automating the majority of your tests, which are fairly straightforward.
* Leave the complex tests for manual testing.
* Never make the mistake of using Record and Playback as a method of automation testing. This method is fraught with problems, and such scripts are very costly to maintain. At best it can be used to perform a Proof of Concept. In the next section, I will elaborate on why I feel so strongly against using this approach.
* As far as possible use a data-driven automated testing methodology. This allows you to develop automated test scripts that are more "generic", requiring only that the input and expected results be updated. This methodology is ideal for testing systems that are huge, complex and have varied test environments. The different data-driven methodologies will be discussed in detail later in this article.
* Most importantly, have a set of resources (hardware, software and people) dedicated to test automation. Resources that share their time across manual and automation testing rarely achieve effective test automation.

Record and Playback (A Test Automation Nightmare)
This methodology alone is probably most responsible for software companies all over the world to give up on test automation out of sheer frustration and return to manual testing. Ironically, this is the tag line that most vendors use to sell their products.

Record/Playback is something that doesn’t work in a real-world scenario and here are the reasons:
* Such scripts contain hard coded values, which must change if anything at all changes in the application. This creates an additional overhead, as the testing team needs to update the scripts each time the data changes.
* These scripts contain very little error handling and recovery scenarios. Consequently, such scripts are not reliable (even in cases where the application doesn’t change) and often fail during playback for a variety of reasons such as pop-up windows, messages etc.
* The cost of maintaining these scripts is huge and quite unacceptable.
* If the application under test changes, the tests must be re-recorded.
* Record/Playback scripts work well only if the application under test is already working as per the expectations. In essence, this defeats the purpose of test automation.
* Record/Playback might work in some cases; if the tools used are intelligent enough to analyze the application under test and auto generate meaningful test data.

{mospagebreak title=Test Automation Strategies}Having discarded Record/Playback as a feasible test automation strategy, we will now move on to some of the more reasonable and widely used test automation strategies.

Figure 1 will explain the different types of test automation.
As Figure 1 indicates, Test Automation is broadly divided into 2 main methodologies.
* Functional Decomposition Method: A system broken down into its components in such a manner that the components correspond directly to system functions and sub functions.
* Key Word Driven Method: A system broken down into certain predefined keywords in such a manner that the keywords correspond directly to system functions and sub functions.

Lets have a look at different approaches to test automation in detail.

Functional Decomposition Method
The main and most important concept behind functional decomposition is to separate the data from the functions. This is achieved by employing a hierarchical architecture with a modular design.

The highest script in the hierarchy is the controller / controlling script. This is the engine of the test automation suite. Typically the Driver script internally calls one or more scripts. These verification scripts are the one’s, which perform the application testing. Apart from this, usually there are a few utility scripts that are used by the driver and verification scripts to perform a number of tasks.
* Controller / Driver Scripts: perform initialization (if required), and then call the business logic verification scripts in the desired order.
* Verification / Test Case Scripts: perform the Business Functions. Usually, the entire business logic is embedded within these scripts. * Subroutine / Utility Scripts: perform application specific tasks required by two or more Business scripts. The generic nature of these scripts can contribute heavily to rapid script development.
* User-Defined Functions: general and/or application-specific functions. Such scripts can be shared across test automation suites.
{mospagebreak title=Functional Decomposition Explained}
Listing 1 will explain the above steps in much better way.

Listing 1:"Login" Test Case for any web based email application

1. Open a web browser.
2. Navigate to the appropriate URL.
3. Verify that the login page is displayed correctly.
4. Write any errors to an Error Log.
5. Retrieve the login details from a data file / database.
6. Enter the user name and password and click on the login button.
7. Verify that the login is correct.
8. Write any errors to an Error Log.
9. Verify the main menu displayed.
10. Write any errors to an Error Log.
11. Logout from the application and close the web browser.

The above steps can be split up effectively into a verification script and couple of subroutines, which can be placed in a library file. For example, the step #5 (retrieves the login details from a data file / database) can be written as a library subroutine, which will have the file / database details as input parameter and return the retrieved values, back to the calling script.

Such a generic routine can be used for data retrieval in numerous other verification scripts. Thus, if it were needed to retrieve test data for 10 different screens, we would need to call only a single subroutine with different input parameters. Having the test data in text / flat files gives an added advantage as updating these files does not require knowledge of any tool, scripting or programming, meaning that the non-technical testers can easily run the tests, whereas the technical resources can create and maintain the test scripts. Hence, maintaining such scripts is very cost effective.

Advantages of using the Functional Decomposition Method:
* A modular design and using files or records to both input and verify data, reduces redundancy and amount of effort in creating automated test scripts.
* Scripts can be developed even when the application development is in progress. If the functionality changes, only the specific verification scripts containing the business logic need to be updated.
* Script reusability is very high. As the scripts are written to test various business functions independently, they can easily be combined in a controlling script in order to accommodate complex test scenarios.
* Maintaining the expected results for such scripts is very easy.
* Error handling is much more robust in these scripts. This allows unattended execution of the test scripts.
* Since such scripts have very little interdependency they can be used in a plug and play manner.

Disadvantages of using the Functional Decomposition Method:
* Technical and skilled personnel are required to create and maintain the scripts.If the test automation suite is very large, then maintaining the expected results can be quite time consuming.
* The testing team needs to maintain both the scripts as well as the test data.
* Proper care should be taken that the test data is maintained in appropriate file formats. If the test data is updated using some non-standard tool, it can cause problems during test execution.

{mospagebreak title=Key Word Driven Approach}

Lets now look at the other important type of test automation, the key word driven method.

Key Word Driven Method:
Totally based on keywords, this method is assists the non-technical resources of the testing team in script development and maintenance. The method uses test case document(s) developed by testing team using a spreadsheet containing special key words. This method preserves most of the advantages of the "Functional Decomposition" method, while eliminating most of the disadvantages.

Full article...

Other Resource

... to read more articles, visit

An Introduction to Software Test Automation