Software QA FYI - SQAFYI

SILKTEST AND WINRUNNER FEATURE DESCRIPTIONS

By: Horwath/Green/Lawler

1. INTRODUCTION

This document provides feature descriptions of two GUI automated test tools and their associated test management software:

• Mercury-Interactive’s WinRunner, version 6.02 (with TestDirector)

• Segue’s SilkTest, version 5.03 (with SilkOrganizer)

The goal of this document is to describe as objectively as possible the capabilities provided by each tool for each feature, rather than trying to state that one tool’s implementation of a feature is better than the other. The feature descriptions made in this document are intentionally terse and condensed—therefore subtle nuances and deep details of a feature are not covered.

This document by itself should only be considered a good starting point for a complete test tool evaluation by those readers who are considering one of these tools for their automated testing efforts. Many excellent evaluation criteria ideas can be found in the AUTOMATED TESTING forum on the www.betasoft.com website. Assigning a qualitative value to each feature is an evaluation task which depends on the test team’s: • testing environment, • programming skills, and the • application testing requirements.

There is no substitute for using each tool to develop a specific set of prototype testcases, in order to determine the strength and weaknesses of each tool against a typical application in your environment.

1.1 Author Backgrounds

The following people developed this report:

• Terry Horwath is the primary author and a test automation consultant. He has worked with SilkTest, QaPartner, and SilkOrganizer since 1994. He has worked for the last year with WinRunner and TestDirector on two large web testing projects and is a TestSuite 6.0 certified product specialist (CPS). Terry can be contacted at thorwath@LakeFolsom.com.

• John Green is a contributing author and owns Automation Expertise Inc., a company which provides customized test automation solutions. He is a recognized expert using SilkTest, QaPartner, and SilkOrganizer, working with these products since 1993. John can be contacted at jwgreen@automationexpertise.com.

• Tom Lawler is a contributing author and is a test automation consultant. He has worked with web, VB, PowerBuilder and Terminal Emulation (TE) applications using WinRunner and TestDirector since 1999. He is a TestSuite 6.0 certified product specialist (CPS) in both GUI and TE and can be contacted at tom.lawler@theballardgroup.net.

2. FEATURE DESCRIPTIONS

For the sake of consistency alphabetical ordering was selected to describe SilkTest features first, followed by WinRunner features in each of the following sections.

2.1 Startup Initialization and Configuration

• SilkTest derives its initial startup configuration settings from its partner.ini file. This though is not important because SilkTest can be reconfigured at any point in the session by either changing any setting in the Options menu or loading an Option Set. An Option Set file (*.opt) permits customized configuration settings to be established for each test project. The project specific Option Set is then be loaded [either interactively, or under program control] prior to the execution of the project’s testcases. The Options menu or an Option Set can also be used to load an include file (*.inc) containing the project’s GUI Declarations [discussed in section 2.6 on page 5], along with any number of other include files containing library functions, methods, and variables shared by all testcases.

• WinRunner derives its initial startup configuration from a wrun.ini file of settings. During startup the user is normally polled [this can be disabled] for the type of addins they want to use during the session [refer to section 2.3 on page 3 for more information about addins].

The default wrun.ini file is used when starting WinRunner directly, while project specific initializations can be established by creating desktop shortcuts which reference a project specific wrun.ini file. The use of customized wrun.ini files is important because once WinRunner is started with a selected set of addins you must terminate WinRunner and restart it to use a different set of addins.

The startup implementation supports the notion of a startup test which can be executed during WinRunner initialization. This allows project-specific compiled modules [memory resident libraries] and GUI Maps [discussed in section 2.6 on page 5] to be loaded. The functions and variables contained in these modules can then be used by all tests that are run during that WinRunner session.

Both tools allow most of the configuration setup established in these files to be over-ridden with runtime code in library functions or the test scripts.

2.2 Test Termination

• SilkTest tests terminate on exceptions which are not explicitly trapped in the testcase. For example if a window fails to appear during the setup phase of testing [i.e. the phase driving the application to a verification point], a test would terminate on the first object or window timeout exception that is thrown after the errant window fails to appear.

• WinRunner tests run to termination [in unattended Batch mode] unless an explicit action is taken to terminate the test early. Therefore tests which ignore this termination model will continue running for long periods of time after a fatal error is encountered. For example if a window fails to appear during the setup phase of testing, subsequent context sensitive statements [i.e. clicking on a button, performing a menu pick, etc.] will fail—but this failure occurs after a multi-second object/window “is not present” timeout expires for each missing window and object. [When executing tests in non-Batch mode, that is in Debug, Verify, or Update modes, WinRunner normally presents an interactive dialog box when implicit errors such as missing objects and windows are encountered].

2.3 Addins and Extensions

Out of the box, under Windows, both tools can interrogate and work with objects and windows created with the standard Microsoft Foundation Class (MFC) library. Objects and windows created using a non-MFC technology [or non-standard class naming conventions] are treated as custom objects. Dealing with truly custom objects is discussed further in section 2.8 on page 6. But objects and windows created for web applications [i.e. applications which run in a browser], Java applications, Visual Basic applications, and PowerBuilder applications are dealt with in a special manner:

• SilkTest enables support for these technologies using optional extensions. Selected extensions are enabled/disabled in the current Option Set [or the configuration established by the default partner.ini option settings].

• WinRunner enables support for these technologies using optional addins. Selected addins are enabled/disabled using either the Addin Manager at WinRunner startup, or by editing the appropriate wrun.ini file prior to startup. Note that (1) some combinations of addins [WinRunner] and extensions [SilkTest] are mutually exclusive, (2) some of these addins/extensions may no longer be supported in the newest releases of the tool, (3) some of these addins/extensions may only support the last one or two releases of the technology [for example version 5 and 6 of Visual Basic] and (4) some of these addins and extensions may have to be purchased at an addition cost.

2.4 Visual Recorders

SilkTest provides visual recorders and wizards for the following activities: • Creating a test frame with GUI declarations for a full application and adding/deleting selective objects and windows in and existing GUI declarations frame file. • Capturing user actions with the application into a test case, using either context sensitive [object relative] or analog [X:Y screen coordinate relative] recording techniques. • Inspecting identifiers, locations and physical tags of windows and objects. • Checking window and object bitmaps [or parts thereof]. • Creating a verification statement [validating one or more object properties].

WinRunner provides visual recorders and wizards for the following activities: • Creating an entire GUI Map for a full application and adding/deleting selective objects and windows in an existing GUI Map. It is also possible to implicitly create GUI Map entries by capturing user actions [using the recorder described next]. • Capturing user actions with the application into a test case, using either context sensitive [object relative] or analog [X:Y screen coordinate relative] recording techniques. • Inspecting logical names, locations and physical descriptions of windows and objects. • Checking window and object bitmaps [or parts thereof]. • Creating a GUI checkpoint [validating one or more object properties]. • Creating a database checkpoint [validating information in a database]. • Creating a database query [extracting information from a database]. • Locating at runtime a missing object referenced in a testcase [and then adding that object to the GUI Map]. • Teaching WinRunner to recognize a virtual object [a bitmap graphic with functionality]. • Creating Data Tables [used to drive a test from data stored in an Excel-like spreadsheet]. • Checking text on a non-text object [using a built-in character recognition capability]. • Creating a synchronization point in a testcase. • Defining an exception handler.

Some of these recorders and wizards do not work completely for either tool against all applications, under all conditions. For example neither tool’s recorder to create a full GUI Map [WinRunner] or test frame [SilkTest] works against large applications, or any web application. Evaluate the recorders and wizards of interest carefully against your applications if these utilities are important to your automated testing efforts.

Full article...


Other Resource

... to read more articles, visit http://sqa.fyicenter.com/art/

SILKTEST AND WINRUNNER FEATURE DESCRIPTIONS