The Rational Approach to Automated Testing
How Can Software Fail?
The advantages of closing the quality gap early in the development cycle are apparent. The first step is to
measure the gap consistently, but few development organizations have the tools, data, methods and
metrics to do so. How can you measure the quality gap? Or put another way, how can software fail?
Essentially software failures can be classified into four dimensions, which correspond to four aspects of
the quality gap:
l Reliability: Does the application operate without crashing, hanging or leaking memory and other
l Functionality: Does the application meet the business requirements established for it?
l Performance: Does the application respond in a timely manner?
l System Performance: Does the application continue to perform correctly and in a timely manner when it is
subjected to production load?
dimension can result in the release of software that the end-user finds useless or even damaging.
Knowing when to look for defects is just as important as knowing where to look for them. Today, the vast
majority of software engineering follows a waterfall process. In this model of the software development
lifecycle, testing phases are sequential. The most common criticism of this approach is that it takes a long
time and is inadequate to represent the actual process of software design, construction and improvement
necessary to support today's complex business systems. 3 However, there is a much more significant
drawback — namely that a great deal of risk exists in the project until the final stage. No one working on
the project can be certain that the system will work until very late in the process — sometimes too late.
Further, no one can accurately assess the project cost until most of the planned budget is spent. And
that's the reason that three of four major software projects undertaken are late or over budget.
Test Early, Test Automatically, Test with Every Iteration
The most effective way to reduce risk is to start testing early in the development cycle and to test
iteratively, with every build. With this approach, defects are removed as the features are implemented.
The testing of the application is completed shortly after the final features are coded, and as a result the
product is ready for release much earlier. Additionally, the knowledge of what features are completed, that
is both coded and tested, affords management greater control over the entire process and promotes
effective execution of the business strategy. Testing with every iteration may require some additional
upfront planning between developers and testers, and a more earnest effort to design for testability; but
these are both inherently positive undertakings, and the rewards are substantial.
There are several key advantages gained by testing early and with every build to close the quality gap
l Risk is identified and reduced in the primary stages of development instead of in the closing stages.
l Repairs to problems are less costly.
l The release date can be more accurately predicted throughout the project.
l The product can be shipped sooner.
l The business strategy can be executed more effectively.
A Unique Solution and Industry Leadership
The goal of Rational Software is to ensure the success of customers who depend on their ability to
develop software. Rational's automated software testing tools are only one part of a unique solution to
achieving customer success. The complete solution is based on providing the user with the principles,
tools, and services needed to efficiently develop software. The principles are Rational's best practices of
software development, gleaned from many years of experience in the industry:
l Develop software iteratively
l Manage requirements
l Employ component-based architectures
l Model software visually
l Verify software quality
l Control changes to software.
From the start, Rational designed a set of interoperable software tools for requirements management,
visual modeling, automated testing, and configuration and change management to enable the customer to
easily and effectively employ these principles. Today, Rational offers a complete set of software
development tools that support test automation and the test workflow.
Rational Suite TestStudio includes these award-winning components in a team unifying, seamlesslyintegrated
l Rational TestFactory™, to detect application crashes without user intervention and generate optimal
scripts for regression testing
l Rational Robot, to provide thorough functional testing across all aspects of the application under test
l Rational Purify®, for locating hard-to-find runtime errors
l Rational Visual PureCoverage™, to identify untested code and provide code-coverage analysi
l Rational Visual Quantify™, to pinpoint performance bottlenecks
l Rational Requisite®Pro, to manage and track requirements
l Rational ClearQuest™, to manage and track change requests and defects
l Rational Unified Process, a comprehensive knowledge base of software development best practices
l Rational SoDA®, for automated reporting and documentation
To close the final dimension of the quality gap, Rational offers Rational PerformanceStudio™, the most
powerful system performance testing tool available.
Rational's services round out this unique solution and provide the user with training, consulting and quick,
complete answers to questions.
This product line has propelled Rational to its current position as a recognized market leader. It has the
highest growth rate in the requirements management market, and is the market share leader in visual
modeling, configuration and change management, testing tools. The entire line of products has won an
unprecedented series of awards and honors from editors and users alike. Consequently, its not surprising
that Rational's customer base includes companies that depend on their ability to produce high quality
an associated set of activities; and the level of effort needed for each workflow varies depending on what
phase the project is in — inception, elaboration, construction, or transition. The requirements workflow
begins in inception and requires significant attention at this stage. In the later stages, as requirements are
adjusted and fine-tuned, less effort is required. Rational's award-winning Rational RequisitePro helps
project managers establish requirements at the outset, and maintain them as they evolve throughout the
life of the project. Because it is integrated with Rational's testing tools, completed features are
automatically tracked against the requirements they fulfill.
The test workflow is further divided into the four dimensions of software quality that must be tested:
reliability, functionality, application performance, and system performance. Reliability testing begins as
soon as there is executable code, and should become part of the compile-debug cycle. Functional testing
begins when a module or subsystem designed to meet a requirement is completed. Application
performance testing begins along with or soon after functional testing begins. System performance testing
typically needs to wait until major functions of the target application are complete, but the testing begins
long before the system needs to be deployed.
Rational Suite TestStudio combines a powerful array of interoperable software and team unifying tools
that enable managers, developers, and testing professionals to work together more efficiently than ever
before. It features Rational's market leading tools for testing application reliability, functionality and
performance. In addition to improving individual productivity, Rational Suite TestStudio breaks down the
barriers that have traditionally existed between groups. It includes software that improves communication
both within and between teams, increasing cooperation while simplifying project coordination.
Rational's comprehensive suite of automated testing software includes powerful tools that support the
best practices of software testing and close every dimension of the quality gap:
l Reliability Testing and Code-coverage Analysis
When an organization is committed to testing early and testing with every iteration throughout
development, automated testing is no longer a luxury, it is a necessity. While many testing tools claim to
be automated, the majority of the tasks associated with these tools are manual. Rational TestFactory is
the only testing tool that automatically and proactively finds defects in the application under test while
building an optimized set of test scripts that maximize code coverage. Using a revolutionary new
technology to deliver on the promise of automated testing, Rational TestFactory first builds a
comprehensive, hierarchical map of the application's user interface. This application map is then used to
automatically generate a set of test scripts that cover the application's user interface and source code to
the greatest extent possible. Rational Suite TestStudio includes Rational Visual PureCoverage,
enabling testers and developers to see immediately what percentage of source code has been exercised
and easily identify untested or insufficiently tested functions, procedures and methods. As Rational
TestFactory exercises an application it automatically detects program defects — everything from serious
defects like hangs, crashes and exceptions, to subtle problems like a dialog box button without an
Rational TestFactory exercises your application more thoroughly than any feasible amount of manual
testing can. The burden of having to re-record or rewrite test scripts frequently as an application goes
through early development is virtually eliminated because Rational TestFactory can easily regenerate any
part of the application map and associated scripts. Rational TestFactory closes the reliability gap, allowing
you to develop a stable application that provides a strong foundation for further requirements-based and
For more in-depth unit testing, Rational Purify can be employed to detect run-time errors and a variety of
memory-access errors, and locate memory and other resource leaks and API calls with invalid
parameters. Both Rational Visual PureCoverage and Rational Purify work within Microsoft Developer
Studio, and both can be driven using Rational's record and playback tool to ensure repeatable tests. By
using patented Object Code Insertion (OCI) technology both tools are able to work on applications for
which no source code is available — thereby providing the only means available to certify third-party
l Functional Testing
After developers implement the project requirements established with Rational RequisitePro, testing
professionals test the application with Rational Robot, the industry's leading record/playback functional
testing tool. Rational Robot's unique Object Testing ™ technology leaves traditional GUI testing far
behind as it tests all of the hundreds or thousands of properties of an object, even if they are invisible.
The scripts that Rational Robot creates reliably detect errors in successive builds of the application while
remaining virtually unaffected by changes to it and they are platform independent. The same Rational
Robot script can test, without modification, the application on multiple Windows platforms. Object Testing
™ delivers the fastest, most intuitive test creation with the shortest learning curve-scripts are easy to
create, and easy to play back. Rational Robot supports a wide range of environments and languages
including Microsoft Visual Basic and Visual C++, Java, HTML, DHTML, Oracle Developer/2000, and
Sybase PowerBuilder, as well as ERP environments from SAP, PeopleSoft, Oracle and others. Rational
Robot's extensive support for Dynamic HTML and Java provides testing professionals with unparalleled
power to test Web and e-commerce applications. All of the test assets created and used by Rational
Robot are stored in a central test repository. Rational TestManager provides direct access to this
repository and includes a powerful report writer and graphing engine that delivers concise easy to
understand summaries of the testing process. Rational TestManager can be used to develop a test planeither
directly, or by importing requirements from Rational RequisitePro-and different parts of the plan can
be immediately assigned to individual developers. In addition, Rational Suite: TestStudio includes
Rational ClearQuest, an e-mail enabled and Web-accessible defect tracking system. Any failed
automated test results are automatically logged in the repository; while notifications of failed tests,
repaired defects, and assigned tasks are automatically sent to appropriate team members via any MAPI
or SMTP compliant e-mail system.
l Application Performance Testing
Like Rational Visual PureCoverage and Rational Purify, Rational Visual Quantify, Rational's application
performance profiling tool, works with Microsoft Developer Studio and works on applications with or
without source code. Rational Visual Quantify tests performance by automatically identifying bottlenecks
in all parts of the application. After Rational Visual Quantify determines where the application spends the
bulk of its time, it enables the tester to determine how it spends it. It can compare time spent in the
application with time spent in system calls, to establish whether the application or the platform is the
primary cause of slow performance.
Rational Visual Quantify, Visual PureCoverage and Purify can all be driven by Rational Robot as it plays
back a script. This ensures that all reliability and application performance tests are repeatable. Once a runtime
error or performance bottleneck has been fixed, the same script that was used to pinpoint the
problem can then be run again to ensure that the problem has been eliminated. Additionally, Rational
Visual PureCoverage can combine the result from several script runs to aggregate code coverage results
for a suite of individual tests.
l System Performance Testing
Rational PerformanceStudio is the most powerful system performance tool available today.
PerformanceStudio delivers unparalleled power and unique capabilities in every step of the system
performance testing process. A number of innovations allow PerformanceStudio to simulate the most
realistic application load possible while collecting all the data needed to measure and accurately predict
system performance exactly as the end user will experience it.
With DataSmart Recording, Rational PerformanceStudio can record a script at the GUI level by working
with Rational Robot, or in secure environments it can record at the API level. It can also record "on the
wire" to produce the most accurate and scalable tests available. After the script is recorded, Rational
PerformanceStudio automatically creates a parameterized version of it, which can be played back using
data sets to simulate hundreds or thousands of users. Rational PerformanceStudio uses ClientSmart
Pacing to accurately replicate the timing of the production environment including user and client delays,
as well as CPU and networking delays. LoadSmart Scheduling offers the ability to simulate different
groups of users in a schedule. Simple point and click operations are all that is needed to create a
schedule that simulates 30% query users, 20% entry users and 50% report users, for example. These
proportions can then be used to simulate a 10-user load or a 5000-user load. The tests can be monitored
as they are running and, if desired, the test engineer can drill down and examine the source code that is
being executed at any time during the test. Lastly, Rational PerformanceStudio uses ServerSmart
Playback to adjust its script playback to facilitate dynamic HTML pages. Rational PerformanceStudio is so
easy to use and configure that a set up time of a few minutes is all that is required to obtain important
system performance metrics that would have been all but impossible to acquire by other means.
The recent changes in the industry have several implications for system performance testing. First,
quicker releases mean that more people will need to be able to create and execute system performance
tests than ever before, so the tools must be easy to use. Second, because the number of users that will
be accessing the server is not easily predicted, scalability in the system performance tool must be easy
an automatic. Finally, accuracy is now a must-have. It is no longer good enough to just "hammer the
server." Testing results must be easily understood to ensure that users will get correct responses in
acceptable time frames. Rational PerformanceStudio is the only system performance testing tool
designed to satisfy the demanding constraints imposed by these implications.
Unifying Software Teams
Rational Suite TestStudio is one just member of the Rational Suite family of products. These powerful
products unify your software development teams, bringing them together as a collaborative force. Each
suite is designed to meet the specific needs of individual managers, engineers and technicians that make
up your teams, while dramatically improving their ability to communicate and share their work with other
team members. The Rational Suite family also includes:
Rational Suite AnalystStudio - combining a proven set of unifying tools and market leading analysis
tools, to create a complete solution for analysts, and
Rational Suite DevelopmentStudio - delivering a comprehensive solution for architects and developers
with a powerful array of team-based, market leading design and development tools.
... to read more articles, visit http://sqa.fyicenter.com/art/