Software QA FYI - SQAFYI

Advanced Risk Base d Test Results Reporting:


Analytical risk based testing offers a number of benefits to test teams and organizations that use this strategy.

One of those benefits is the opportunity to make risk-aware release decisions. However, this benefit requires risk based test results reporting, which many organizations have found particularly challenging. This article describes the basics of risk based testing results reporting, then shows how Rex Black (of RBCS ) and Nagata Atsushi (of Sony) developed and implemented new and ground-breaking ways to report test results based on risk.

Analytical Risk Based Testing Testing can be thought of as (one) way to reduce the risks to system quality prior to release. Quality risks typically include possible situations like slow system response to use input, incorrect calculations, corruption of customer data, and difficulty in understanding system interfaces. All testing strategies,competently executed, will reduce quality risks. However, analytical risk based testing, a strategy that allocates testing effort and sequences test execution based on risk, minimizes the level of residual quality risk for any given amount of testing effort. There are various techniques for risk based testing, including highly formal techniques like Failure Mode and Effect Analysis (FMEA). Most organizations find this technique too difficult to implement, so RBCS typically recommends—and helps clients to implement—a technique called Pragmatic Risk Analysis and Management (PRAM).

In analytical risk based testing,stakeholder interviews, requirements specifications, past defect history, and other sources of information are used to develop a categorized list of quality risk items, which are then assessed to determine, for each risk item, the likelihood of bugs related to the risk item and the impact of such bugs should they exist in the system after release. Impact is typically determined from a business perspective, while likelihood involves consideration of technical issues. Likelihood and impact are then combined to determine an overall rating of risk for each risk item (see Rex Black’s book Managing the Testing Process, 3e, for a detailed description of the process of quality risk analysis).

This catalog of prioritized risk items is then used to develop and execute tests. Tests are developed for each risk item, with the precise number of tests covering each risk item based on the level of risk. As tests are developed, they are given a priority based on the priority of the risk item they cover.

Then, during test execution, tests are run in risk priority order. This provides three immediate benefits:

n Tests effort is tightly calibrated to the level of risk reduction that any given functional or non-functional attribute requires.
n Tests are run in an order that maximizes the chances of discovering the most important bugs early in test execution.
n If necessary due to schedule pressures, less important tests (which are sequenced towards the end of test execution) can be eliminated.
However, if, during test development and execution, the test team uses a test management tool, a database, or even a simple spreadsheet to maintain traceability between risk items, the tests that cover them, the results of those tests, and bugs found during testing, the organization gains a fourth benefit: the ability to make fully-informed, risk-aware decisions about whether to release software based on the residual level of quality risk.
This benefit is of great importance, because a large number of project teams make release decisions without a full understanding of the current quality status of the system under test. This is because the standard test management reporting dashboards, based on charts and tables showing bug status and test pass/fail status, are at best indirect and imperfect measures of quality and quality risk. Like a shadow thrown by a candle in a dark room, the picture given by such charts and graphs is flickering, unclear in the details, and distorted. Risk based results reporting, when done properly, allows everyone, testers and non-testers alike, to see a clear, direct, steady picture of quality risk.

Basic Risk Based Test Results Reporting How does risk based test results reporting work? There are three basic approaches.

The first approach, and simplest, approach relies on reorganizing the traditional metrics of testing, bug status and test pass/fail status, based on their relationship to risks. An example of this kind of reporting is shown in Table 1, using a hypothetical e-commerce application.

Full article...

Other Resource

... to read more articles, visit

Advanced Risk Base d Test Results Reporting: