Testing GUI Applications
By: Paul Gerrard
Most clients in client/server systems deliver system functionality using a graphical user interface (GUI). When testing complete systems, the tester must grapple with the additional functionality provided by the GUI. GUIs make testing systems more difficult for many reasons: the event-driven nature of GUIs, unsolicited events, many ways in/many ways out and the infinite input domain problems make it likely that the programmer has introduced errors because he could not test every path.
Available literature on testing GUIs tends to focus on tools as the solution to the GUI testing problem. With few exceptions, papers on this topic that have been presented at EuroSTAR and STAR (in the US) over the last few years have paid little attention to GUI test design but have concentrated on how automated regression-test suites can be built and maintained. Our intention is to attempt to formulate a GUI test strategy to detect errors and use tools to assist this process. If we achieve this, the task of building regression test suites will be made much easier.
This paper describes the GUI testability ‘problem’ and offers guidelines on testing these sticky objects manually and with automated tools. We present a summary of the types of error that occur most often in GUI applications and propose an approach to designing GUI tests that focus on these errors. The approach consists of a series of test types that are described in turn. These techniques are assembled into a GUI specific test process that can be mapped onto organisations’ existing staged test process.
This is an interim paper based on ongoing research and work on client projects.
Prerequisite Key Words: none
Topic Descriptors: Testing, GUI, Graphical User Interfaces
1.1 GUIs as universal client
GUIs have become the established alternative to traditional forms-based user interfaces. GUIs are the assumed user interface for virtually all systems development using modern technologies. There are several reasons why GUIs have become so popular:
* GUIs provide the standard look and feel of a client operating system.
* GUIs are so flexible that they can be used in most application areas.
* The GUI provides seamless integration of custom and package applications.
* The user has a choice of using the keyboard or a mouse device.
* The user has a more natural interface to applications: multiple windows can be visible simultaneously, so user understanding is improved.
* The user is in control: screens can be accessed in the sequence the user wants at will.
1.2 GUIs v forms
Lets look at the differences between GUIs and forms based interfaces.
Forms Based Applications
In forms-based applications, the forms are arranged in a hierarchical order. Most often, a top-level menu is displayed which offers a selection of options and when one option is chosen, the selected screen is displayed. Often, one menu calls another lower level menu to provide a further level of selection. In large applications or packages, there might be three levels of menus to be navigated, before the required functionality is presented to the user. A two level menu system containing fifteen options per menu can provide rapid access to over two hundred screens.
With forms displayed on a screen, we can only display and therefore interact with one form at a time. Usually, when a new form is displayed, it fills the screen and functionality on the old form is now unavailable. The one-at-a-time mode is the main characteristic of forms-based systems. In some applications navigation is achieved form-to form, by providing command-driven interfaces. In this way, the user avoids having to navigate using the menu system.
Typically, expert users use such command-driven methods, while occasional users adopt the menu-driven approach. In a sophisticated application, expert users may navigate to virtually any system feature from any other, as long as they know what commands are available, wherever they are. Occasional users are left with the problem of always having to navigate through menus. In large, complex systems, this can be a major headache.
In application forms, the fields on the form have a predefined and unchangeable ‘tabbing order’. That is, the user may only access the fields in a certain order, regardless of whether any data has been entered into the form fields. Navigation is normally achieved through use of the tab key to move forwards and the backspace key to go backwards.
The most obvious characteristic of GUI applications is the fact that the GUI allows multiple windows to be displayed at the same time. Displayed windows are ‘owned’ by applications and of course, there may be more than one application active at the same time.
Access to features of the systems is provided via three mechanisms. Menu bars provide almost continuous availability of the various features of the systems; buttons and keyboard shortcuts enable the user to navigate and access the various functions of their application.
Windows provide forms-like functionality with fields in which text or numeric data can be entered. But GUIs introduce additional objects such as radio buttons, scrolling lists, check boxes and other graphics that may be displayed or directly manipulated.
The GUI itself manages the simultaneous presentation of multiple applications and windows. Hidden windows in the same or different applications may be brought forward and used. There are few, if any, constraints on the order in which users access GUI windows so users are free to use the features of the system in the way they prefer, rather than the way the developers architected it.
Fields within windows have a tabbing order, but the user is free to use the mouse to change the focus of the application to any field on screen. There are no constraints on the order in which a user may enter data on a screen. To the user, there are advantages in being able to access fields directly (perhaps to avoid tabbing through many fields that will not change).
In short, GUIs free the user to access system functionality in their preferred way. They have permanent access to all features and may use the mouse, the keyboard or a combination of both to have a more natural dialogue with the system.
1.3 Some testing difficulties
GUIs have brought considerable benefits to developers. They release the developer from the concerns of interface design – in most environments, GUI design standards impose conventions which make one application look very much like another on the same platform.
However, the sophistication and simplicity of a GUI hides the complexity from the user and where development frameworks are used, the programmers too. When testers are presented with a GUI application to test, the hidden complexities become all too obvious. Consequently, testing GUIs is made considerably more difficult. What are the reasons for this?
The event-driven nature of GUIs presents the first serious testing difficulty. Because users many click on any pixel on the screen, there are many, many more possible user inputs that can occur. The user has an extremely wide choice of actions. At any point in the application, the users may click on any field or object within a window. They may bring another window in the same application to the front and access that. The window may be owned by another application. The user may choose to access an operating system component directly e.g. a system configuration control panel.
The large number of available options mean that the application code must at all times deal with the next event, whatever it may be. In the more advanced development environments, where sophisticated frameworks are being used, many of these events are handled ‘behind the scenes’. With less advanced toolkits, the programmer must write code to handle these events explicitly. Many errors occur because the programmer cannot anticipate every context in which their event handlers are invoked.
Many events such as button clicks cause the focus of the application to move from one feature to another completely unrelated feature. Not only does the selected feature have to deal with a potentially unknown context, the previous feature may be ‘left hanging’ in a partially completed state. The number of potential paths from feature to feature within the application is so high that the scope for programmers to make errors is dramatically increased. The ‘infinite paths’ problem also makes it extremely unlikely that they will all be tested.
Unsolicited events cause problems for programmers and testers. A trivial example would be when a local printer goes off-line, and the operating system puts up a dialog box inviting the user to feed more paper into the printer. A more complicated situation arises where message-oriented middleware might dispatch a message (an event) to remind the client application to redraw a diagram on screen, or refresh a display of records from a database that has changed.
Unsolicited events may occur at any time, so again, the number of different situations that the code must accommodate is extremely high. Testing of unsolicited events is difficult because of the number of test cases may be high but also special test drivers may be necessary to generate such events within the operating systems.
GUIs map very well to the object-oriented paradigm. The desktop, windows and other graphical elements are usually organised into a hierarchy of objects that deal with GUI events. Every object has its own methods (event handlers) and attributes. Typical attributes define the object’s state and usually include:
* Is the object active or inactive (reacts to mouse clicks to take the focus)?
* Appearance e.g. font type, font size, position on screen, dimensions, visible/invisible, colour.
* Contents e.g. text, on/off, true/false, number and values of entries in a list box.
The number of attributes of objects on screen is large. Even for simple text boxes there may be thirty or more attributes. For the majority of objects, these attributes are static: the appearance attributes are defined once and do not change. However, screen objects that contain data entered by the user must accommodate their changing contents but may also have their own event handlers which may perform validation tasks.
Hidden synchronisation and dependencies
It is common for window objects to have some form of synchronisation implemented. For example, if a check box is set to true, a text box intended to accept a numeric value elsewhere in the window may be made inactive or invisible. If a particular radio button is clicked, a different validation rule might be used for a data field elsewhere on the window.
Synchronisation between objects need not be restricted to objects in the same window. For example, a visible window may present customer details including say, ‘date of last order’. Another open window might be used to log customer orders, so if the user creates and confirms an order for the same customer, should the ‘date of last order’ field on the first window be updated? Most users would suggest it should be so the programmer must use the event handling mechanisms to implement the synchronisation functionality. The problem for the tester is ‘where are these dependencies?’
‘Infinite’ input domain
On any GUI application, the user has complete freedom to click with the mouse-pointing device anywhere on the window that has the focus. Although objects in windows have a default tab order, the user may choose to enter data values by clicking on an object and then entering data. In principle, there may be 5,040 different sequences of entering data into seven data fields (this is seven factorial or 7x6x5x4x3x2x1). For more complex screens the numbers grow dramatically. Do we care as testers, through?
Consider the synchronisation situation above (the selected check box and greyed out numeric field). If the user forgets to check the check box, but then proceeds to enter a value in the numeric field, what should happen if the user then remembers to click on the check box? Should the numeric field be cleared? If not, is the data in numeric field written to the database?
Many ways in, many ways out
An obvious consequence of the event-driven nature of GUIs is that for most situations in the application, there may be ‘many ways in’ by which the user reached that point in the application. How many ways in are there? Should they all be tested? In the majority of situations in GUI applications, there may be ‘many ways out’ also - the user may use a keyboard shortcut, a button click, a menu option, click on another window etc. How many of these should be tested?
For most options within GUI applications, there are three ways of selecting options or implementing functionality: these are keyboard shortcuts, function keys, and mouse movements (buttons or menus). Given that these three mechanisms are available for many options for most of the time, does this mean we must test these features three times over?
In a GUI environment, users take the standard features of window management and control for granted. These features include window movement, resizing, maximisation, minimisation and closure. These are usually implemented by standard buttons and keyboard commands available on every window. The programmer has control over which standard window controls are available, but although the operating system handles the window’s behaviour, the programmer must handle the impact on the application.
In some circumstances, closing a window before completing a transaction may leave the application or the database in an inconsistent state. The programmer might avoid such complications by disabling all of the standard window buttons and commands. But he might also have made it impossible for the user to reverse or undo certain actions. From the tester’s point of view, which standard window controls need to be tested? Where is the dividing line between testing the application and testing the operating system? Do we need to test navigation paths both forwards and backwards?
2 GUI Test Strategy
2.1 Test Principles Applied to GUIs
Our proposed approach to testing GUIs is guided by several principles, most of which should be familiar. By following these principles we will develop a test process which is generally applicable for testing any GUI application. Note that the proposed test approach does not cover white-box testing of application code in any depth. This approach concentrates on GUI errors and using the GUI to exercise tests so is very-oriented toward black-box testing.
Focus on errors to reduce the scope of tests
We intend to categorise errors into types and design test to detect each type of error in turn. In this way, we can focus the testing and eliminate duplication.
Separation of concerns (divide and conquer)
By focusing on particular types of error and designing test cases to detect those errors, we can break up the complex problem into a number of simpler ones.
Test design techniques where appropriate
Traditional black box test techniques that we would use to test forms based applications are still appropriate.
Layered and staged tests
We will organise the test types into a series of test stages. The principle here is that we bring tests of the lowest level of detail in components up front. We implement integration tests of components and test the integrated application last. In this way, we can build the testing up in trusted layers.
Test automation...wherever possible
Automation most often fails because of over-ambition. By splitting the test process into stages, we can seek and find opportunities to make use of automation where appropriate, rather than trying to use automation everywhere.
2.2 High Level Test Process
An outline test process is presented in Figure 1 - The high-level test process. We can split the process into three overall phases: Test Design, Test Preparation and Test Execution. In this paper, we are going to concentrate on the first stage: Test Design, and then look for opportunities for making effective use of automated tools to execute tests.
For more, go to http://www.evolutif.co.uk/GUI/TestGui.html
... to read more articles, visit http://sqa.fyicenter.com/art/