<< Testing Metrics for Developers | Other Metrics for Testing and Development >>

Testing Metrics for Testers

<< Testing Metrics for Developers | Other Metrics for Testing and Development >>

Test Metrics for Testers

An obvious measure of testing effectiveness is how many defects are found the more the better.

But this is not a comparative measure.

You could measure the defects a particular phase of testing finds as a proportion of the total
number of defects in the product. The higher the percentage the more effective the testing. But
how many defects are in the product at any given time? If each phase introduces more defects, this
is a moving target. And how long are you going to wait for your customers to find all the defects
you missed in testing? A month? A year?

And suppose that the developers write software that has little or no defects? This means you will
find little or no defects. Does that mean your testing is ineffective? Probably not, there are simply
less defects to find than in a poor software product.

Instead, you could measure the performance of individual testers.

In a 'script-heavy' environment measures of test efficiency are easy to gather. The number of test
cases or scripts a tester prepares in an hour could be considered a measure of his or her
productivity during preparation. Similarly the total number executed during a day could be
considered a measure of efficiency in test execution.

But is it really?

Consider a script-light or no-script environment. These testers don't script their cases so how can
you measure their efficiency? Does this mean that they can't be efficient? I would argue they can.
And what if the tests find no defects? Are they really efficient, no matter how many are written?

Let's return to the purpose of testing to identify and remove defects in software.

This being the case, an efficient tester finds and removes defects more quickly than an inefficient
one. The number of test cases is irrelevant. If they can remove as many defects without scripting,
then the time spent scripting would be better used executing tests, not writing them.

So the time taken to find a defect is a direct measure of the effectiveness of testing.

Measuring this for individual defects can be difficult. The total time involved in finding a defect may
not be readily apparent unless individuals keep very close track of the time they spend on testing
particular functions. In script-heavy environments you also have to factor in the time taken
scripting for a particular defect, which is another complication.

But to measure this for a particular test effort is easy simply divide the total number of hours
spent in testing by the total number of defects logged (remember to include preparation time).

Defect Discovery Rate = Number of defects found

tester hours

Note that you should only count defects that have been fixed in these equations.

Why?

New defects that haven't been validated are not defects. Defects which are rejected are also not
defects. A defect which has been fixed, is definitely an error that need to be corrected. If you count
the wrong numbers, you're going to draw the wrong conclusions.