Choose QA Metrics around Business Goals
By: Claude Fenner
Metrics are all about creating actions around business goals. When it comes to metrics, there are numerous quality and testing possibilities. It is therefore important to carefully select ones that truly suit your situation. The ones that I have found most useful and cost-effective are discussed below.
Before committing to using any metric, always be prepared to perform the following five steps. If you are not willing to do all of them, then forget about using the metric.
Find data sources and Collect data
Metrics dont produce business value unless they are related to legitimate business goals. If for example customer satisfaction is your business goal, then a metric that measures customer defects is a good choice and a programmer productivity metric is a poor choice.
Once you have a specific metric in mind, be certain you have known, clean data sources to support it. Collect data and then apply the metric to transform the data into information that reveals trends for decision making. Before committing to the metric, think through possible range of data values and determine in advance the decision criteria that will make sense for your business goals.
Finally, only commit to a metric if you are prepared to take actions based on the trend information. Metrics always require overhead. If you are not willing and able to take action, then dont bother wasting your teams energy on the metric. As you will see, there are many excellent metrics cited below. Some of them however are not good choices for many organizations, simply because acting on them is too expensive.
Three categories of metrics that are driven by business goals are
customer satisfaction, asset preservation, and
process efficiency and improvement
Here is a list of proven metrics grouped according to the business purpose they serve. Afterwards, a case study will reveal the subset that in my experience is most often useful and economical.
Escaped Defects Customer reported defects that have escaped all software quality processes are represented in this metric.
Performance and Load The performance and load capabilities of the software are captured by this metric. The trend chart often includes a benchmark of predicted future customer performance requirements.
Scenario This metric represents test coverage of customer-specific configuration, profile-based, or end-to-end scenarios.
Asset Decay Based on the inventory of current software assets, this metric rates what part of them are not operational, non-conforming to maintenance standards, or otherwise degraded.
Asset vs. Demand Assets require investment to build and maintain them. This metric measures the trend of asset inventory against predicted future needs. This metric is particularly useful for budget justification.
Process Efficiency and Improvement
Completeness Using a baseline of tests required for the release, this metric tracks the current level and trend of actual testing.
Open Defects Internally reported defects are represented in this metric. Usually it is summarized in the form of a trend chart and is subdivided by severity.
Stability A trend metric that shows discovery rates as the release is progressing. A declining rate presumably indicates quality is improving.
Defect Classification The origin of each defect is classified according to its underlying process step or originating organization. Used for pinpointing repeating points of failure in the process.
Defect Correction Success This metric indicates what percentage of defects are repaired incorrectly. Used for process improvement.
Personnel Productivity For management interested in measuring and possibly increasing productivity, these metrics measure productivity of team members using tests as units of work.
Schedule Trend The metric shows a preplanned trajectory on a timeline and tracking using actual data. Used for project management.
The following case study is based on my experiences at a company exceedingly committed to quality. The QA metrics were applied over many years, which enabled me to assess their long-term value to the business. The team structure included an offshore organization.
Challenges and Solutions
Escaped Defects Customer reported defects were always tracked. Cleanliness of the data was important here. Technical support screened the defects and removed those that were questions or usage errors.
Performance and Load Tracked the load in a test configuration against a predicted benchmark trend.
Scenario A focused, customer-specific set of configuration and end-to-end scenarios were specified and tracked.
Asset Decay Sizable amounts of code and tests were shipped offshore. Decay was steadily occurring, though nobody noticed it at first because no measurement methodology was in place.
Asset vs. Demand Modeling of asset requirement growth was undertaken every year in conjunction with annual budgeting.
Process Efficiency and Improvement
Completeness This was one of the most useful QA metrics especially when combined with the open defects metric. The trend information was always useful early on during a release, but diminished in reliability as the release date approached. Slowing of the trend line was simply a consequence of lock stepping with development during final defect correction. Nevertheless, the metric overall was very valuable.
Open Defects This was another powerful metric. The challenge here was to keep the defect data clean and relevant. Generally we found that the severity classifications were fine. However, defect data needed to be normalized according to its relevance to customers and release objectives. Defects that could be safely deferred had to be marked so that they didnt appear in the current metric. The house keeping overhead required for this metric was significant but worthwhile.
Stability We discovered that stability at the end of release was based more on stopping testing than the absence of defects. Therefore monitoring the test completion and open defect metrics proved to be just as relevant as this metric but cheaper.
Defect Classification The defect reporting system was enhanced to capture defect classification data. The biggest challenge was to find ways of getting the organization to respond to the conclusions.
Defect Correction Success Data collection was fairly simple. The trend showed the success rate to be steady. As with defect classification, the challenge was to find ways of getting the organization to respond for the sake of improvement.
Personnel Productivity This well-intentioned metric was ultimately a challenge to sell to the QA organization. It appeared to them to be intrusive and communicated a sense of mistrust on the part of the management team.
Schedule Trend The schedule trend metric makes sense to management because it shows a preplanned trajectory and tracking using actual data. The main challenge was always to construct a realistic projected trend line. Once actual data started appearing, the extrapolations were always suspect. Many factors at the end of the release could suddenly throw off an otherwise promising looking curve.
... to read more articles, visit http://sqa.fyicenter.com/art/