Software QA FYI - SQAFYI

The Rational Approach to Automated Testing


How Can Software Fail?
The advantages of closing the quality gap early in the development cycle are apparent. The first step is to measure the gap consistently, but few development organizations have the tools, data, methods and metrics to do so. How can you measure the quality gap? Or put another way, how can software fail? Essentially software failures can be classified into four dimensions, which correspond to four aspects of the quality gap:
l Reliability: Does the application operate without crashing, hanging or leaking memory and other resources?
l Functionality: Does the application meet the business requirements established for it?
l Performance: Does the application respond in a timely manner?
l System Performance: Does the application continue to perform correctly and in a timely manner when it is subjected to production load?

dimension can result in the release of software that the end-user finds useless or even damaging. Knowing when to look for defects is just as important as knowing where to look for them. Today, the vast majority of software engineering follows a waterfall process. In this model of the software development lifecycle, testing phases are sequential. The most common criticism of this approach is that it takes a long time and is inadequate to represent the actual process of software design, construction and improvement necessary to support today's complex business systems. 3 However, there is a much more significant drawback — namely that a great deal of risk exists in the project until the final stage. No one working on the project can be certain that the system will work until very late in the process — sometimes too late. Further, no one can accurately assess the project cost until most of the planned budget is spent. And that's the reason that three of four major software projects undertaken are late or over budget.

Test Early, Test Automatically, Test with Every Iteration
The most effective way to reduce risk is to start testing early in the development cycle and to test iteratively, with every build. With this approach, defects are removed as the features are implemented. The testing of the application is completed shortly after the final features are coded, and as a result the product is ready for release much earlier. Additionally, the knowledge of what features are completed, that is both coded and tested, affords management greater control over the entire process and promotes effective execution of the business strategy. Testing with every iteration may require some additional upfront planning between developers and testers, and a more earnest effort to design for testability; but these are both inherently positive undertakings, and the rewards are substantial. There are several key advantages gained by testing early and with every build to close the quality gap quickly:
l Risk is identified and reduced in the primary stages of development instead of in the closing stages.
l Repairs to problems are less costly.
l The release date can be more accurately predicted throughout the project.
l The product can be shipped sooner.
l The business strategy can be executed more effectively.

Why Rational?
A Unique Solution and Industry Leadership
The goal of Rational Software is to ensure the success of customers who depend on their ability to develop software. Rational's automated software testing tools are only one part of a unique solution to achieving customer success. The complete solution is based on providing the user with the principles, tools, and services needed to efficiently develop software. The principles are Rational's best practices of software development, gleaned from many years of experience in the industry:
l Develop software iteratively
l Manage requirements
l Employ component-based architectures
l Model software visually
l Verify software quality
l Control changes to software.
From the start, Rational designed a set of interoperable software tools for requirements management, visual modeling, automated testing, and configuration and change management to enable the customer to easily and effectively employ these principles. Today, Rational offers a complete set of software development tools that support test automation and the test workflow.

Rational Suite TestStudio includes these award-winning components in a team unifying, seamlesslyintegrated package:
l Rational TestFactory™, to detect application crashes without user intervention and generate optimal scripts for regression testing
l Rational Robot, to provide thorough functional testing across all aspects of the application under test
l Rational Purify®, for locating hard-to-find runtime errors
l Rational Visual PureCoverage™, to identify untested code and provide code-coverage analysi
s l Rational Visual Quantify™, to pinpoint performance bottlenecks
l Rational Requisite®Pro, to manage and track requirements
l Rational ClearQuest™, to manage and track change requests and defects
l Rational Unified Process, a comprehensive knowledge base of software development best practices
l Rational SoDA®, for automated reporting and documentation
To close the final dimension of the quality gap, Rational offers Rational PerformanceStudio™, the most powerful system performance testing tool available.

Rational's services round out this unique solution and provide the user with training, consulting and quick, complete answers to questions.

This product line has propelled Rational to its current position as a recognized market leader. It has the highest growth rate in the requirements management market, and is the market share leader in visual modeling, configuration and change management, testing tools. The entire line of products has won an unprecedented series of awards and honors from editors and users alike. Consequently, its not surprising that Rational's customer base includes companies that depend on their ability to produce high quality software.

an associated set of activities; and the level of effort needed for each workflow varies depending on what phase the project is in — inception, elaboration, construction, or transition. The requirements workflow begins in inception and requires significant attention at this stage. In the later stages, as requirements are adjusted and fine-tuned, less effort is required. Rational's award-winning Rational RequisitePro helps project managers establish requirements at the outset, and maintain them as they evolve throughout the life of the project. Because it is integrated with Rational's testing tools, completed features are automatically tracked against the requirements they fulfill.

The test workflow is further divided into the four dimensions of software quality that must be tested: reliability, functionality, application performance, and system performance. Reliability testing begins as soon as there is executable code, and should become part of the compile-debug cycle. Functional testing begins when a module or subsystem designed to meet a requirement is completed. Application performance testing begins along with or soon after functional testing begins. System performance testing typically needs to wait until major functions of the target application are complete, but the testing begins long before the system needs to be deployed.

Rational Suite TestStudio combines a powerful array of interoperable software and team unifying tools that enable managers, developers, and testing professionals to work together more efficiently than ever before. It features Rational's market leading tools for testing application reliability, functionality and performance. In addition to improving individual productivity, Rational Suite TestStudio breaks down the barriers that have traditionally existed between groups. It includes software that improves communication both within and between teams, increasing cooperation while simplifying project coordination.

Rational's comprehensive suite of automated testing software includes powerful tools that support the best practices of software testing and close every dimension of the quality gap:
l Reliability Testing and Code-coverage Analysis
When an organization is committed to testing early and testing with every iteration throughout development, automated testing is no longer a luxury, it is a necessity. While many testing tools claim to be automated, the majority of the tasks associated with these tools are manual. Rational TestFactory is the only testing tool that automatically and proactively finds defects in the application under test while building an optimized set of test scripts that maximize code coverage. Using a revolutionary new technology to deliver on the promise of automated testing, Rational TestFactory first builds a comprehensive, hierarchical map of the application's user interface. This application map is then used to automatically generate a set of test scripts that cover the application's user interface and source code to the greatest extent possible. Rational Suite TestStudio includes Rational Visual PureCoverage, enabling testers and developers to see immediately what percentage of source code has been exercised and easily identify untested or insufficiently tested functions, procedures and methods. As Rational TestFactory exercises an application it automatically detects program defects — everything from serious defects like hangs, crashes and exceptions, to subtle problems like a dialog box button without an accelerator key.

Rational TestFactory exercises your application more thoroughly than any feasible amount of manual testing can. The burden of having to re-record or rewrite test scripts frequently as an application goes through early development is virtually eliminated because Rational TestFactory can easily regenerate any part of the application map and associated scripts. Rational TestFactory closes the reliability gap, allowing you to develop a stable application that provides a strong foundation for further requirements-based and

performance testing.
For more in-depth unit testing, Rational Purify can be employed to detect run-time errors and a variety of memory-access errors, and locate memory and other resource leaks and API calls with invalid parameters. Both Rational Visual PureCoverage and Rational Purify work within Microsoft Developer Studio, and both can be driven using Rational's record and playback tool to ensure repeatable tests. By using patented Object Code Insertion (OCI) technology both tools are able to work on applications for which no source code is available — thereby providing the only means available to certify third-party components.

l Functional Testing
After developers implement the project requirements established with Rational RequisitePro, testing professionals test the application with Rational Robot, the industry's leading record/playback functional testing tool. Rational Robot's unique Object Testing ™ technology leaves traditional GUI testing far behind as it tests all of the hundreds or thousands of properties of an object, even if they are invisible. The scripts that Rational Robot creates reliably detect errors in successive builds of the application while remaining virtually unaffected by changes to it and they are platform independent. The same Rational Robot script can test, without modification, the application on multiple Windows platforms. Object Testing ™ delivers the fastest, most intuitive test creation with the shortest learning curve-scripts are easy to create, and easy to play back. Rational Robot supports a wide range of environments and languages including Microsoft Visual Basic and Visual C++, Java, HTML, DHTML, Oracle Developer/2000, and Sybase PowerBuilder, as well as ERP environments from SAP, PeopleSoft, Oracle and others. Rational Robot's extensive support for Dynamic HTML and Java provides testing professionals with unparalleled power to test Web and e-commerce applications. All of the test assets created and used by Rational Robot are stored in a central test repository. Rational TestManager provides direct access to this repository and includes a powerful report writer and graphing engine that delivers concise easy to understand summaries of the testing process. Rational TestManager can be used to develop a test planeither directly, or by importing requirements from Rational RequisitePro-and different parts of the plan can be immediately assigned to individual developers. In addition, Rational Suite: TestStudio includes Rational ClearQuest, an e-mail enabled and Web-accessible defect tracking system. Any failed automated test results are automatically logged in the repository; while notifications of failed tests, repaired defects, and assigned tasks are automatically sent to appropriate team members via any MAPI or SMTP compliant e-mail system.

l Application Performance Testing
Like Rational Visual PureCoverage and Rational Purify, Rational Visual Quantify, Rational's application performance profiling tool, works with Microsoft Developer Studio and works on applications with or without source code. Rational Visual Quantify tests performance by automatically identifying bottlenecks in all parts of the application. After Rational Visual Quantify determines where the application spends the bulk of its time, it enables the tester to determine how it spends it. It can compare time spent in the application with time spent in system calls, to establish whether the application or the platform is the primary cause of slow performance.

Rational Visual Quantify, Visual PureCoverage and Purify can all be driven by Rational Robot as it plays back a script. This ensures that all reliability and application performance tests are repeatable. Once a runtime error or performance bottleneck has been fixed, the same script that was used to pinpoint the

problem can then be run again to ensure that the problem has been eliminated. Additionally, Rational Visual PureCoverage can combine the result from several script runs to aggregate code coverage results for a suite of individual tests.

l System Performance Testing
Rational PerformanceStudio is the most powerful system performance tool available today. PerformanceStudio delivers unparalleled power and unique capabilities in every step of the system performance testing process. A number of innovations allow PerformanceStudio to simulate the most realistic application load possible while collecting all the data needed to measure and accurately predict system performance exactly as the end user will experience it.

With DataSmart Recording, Rational PerformanceStudio can record a script at the GUI level by working with Rational Robot, or in secure environments it can record at the API level. It can also record "on the wire" to produce the most accurate and scalable tests available. After the script is recorded, Rational PerformanceStudio automatically creates a parameterized version of it, which can be played back using data sets to simulate hundreds or thousands of users. Rational PerformanceStudio uses ClientSmart Pacing to accurately replicate the timing of the production environment including user and client delays, as well as CPU and networking delays. LoadSmart Scheduling offers the ability to simulate different groups of users in a schedule. Simple point and click operations are all that is needed to create a schedule that simulates 30% query users, 20% entry users and 50% report users, for example. These proportions can then be used to simulate a 10-user load or a 5000-user load. The tests can be monitored as they are running and, if desired, the test engineer can drill down and examine the source code that is being executed at any time during the test. Lastly, Rational PerformanceStudio uses ServerSmart Playback to adjust its script playback to facilitate dynamic HTML pages. Rational PerformanceStudio is so easy to use and configure that a set up time of a few minutes is all that is required to obtain important system performance metrics that would have been all but impossible to acquire by other means. The recent changes in the industry have several implications for system performance testing. First, quicker releases mean that more people will need to be able to create and execute system performance tests than ever before, so the tools must be easy to use. Second, because the number of users that will be accessing the server is not easily predicted, scalability in the system performance tool must be easy an automatic. Finally, accuracy is now a must-have. It is no longer good enough to just "hammer the server." Testing results must be easily understood to ensure that users will get correct responses in acceptable time frames. Rational PerformanceStudio is the only system performance testing tool designed to satisfy the demanding constraints imposed by these implications.

Unifying Software Teams
Rational Suite TestStudio is one just member of the Rational Suite family of products. These powerful products unify your software development teams, bringing them together as a collaborative force. Each suite is designed to meet the specific needs of individual managers, engineers and technicians that make up your teams, while dramatically improving their ability to communicate and share their work with other team members. The Rational Suite family also includes: Rational Suite AnalystStudio - combining a proven set of unifying tools and market leading analysis tools, to create a complete solution for analysts, and

Rational Suite DevelopmentStudio - delivering a comprehensive solution for architects and developers with a powerful array of team-based, market leading design and development tools.

Other Resource

... to read more articles, visit

The Rational Approach to Automated Testing