Software QA FYI - SQAFYI

A Framework for Good Enough Testing

By: James Bach

GOOD ENOUGH TESTING DEFINED In any situation, Good Enough testing asks, How do I know if Im doing, or have done, enough of the right type of testing? Unfortunately, there is no objective or rigorous calculus for answering this question, but we can identify what to consider in attempting to answer it. We can begin to build at least a heuristic framework around the problem. In fact, my general model of Good Enough quality could apply to Good Enough testing as well. Im borrowing some elements from it, but here Im proposing a more specific framework.

As a first step, Ill define the term:
Good Enough testing is the process of developing a sufficient assessment of quality, at a reasonable cost, to enable wise and timely decisions to be made concerning the product.

Let me unpack this definition. Im asserting that, at a high level, its useful to think about the value of testing as a dynamic with four parts:
Assessment of product quality (How accurate and complete is it?).
Cost of testing (How reasonable is it? Is it within project constraints? Is there a good return on investment, such as the extent of information gained per test?).
Decisions (How well does the assessment serve the project and the business?).
Timing of all the above (Is it soon enough to be useful?).
Testing also supports business processes beyond decision-making. But for this short column, I am lumping under decisions the uses any testing client would have for test results, such as creating accurate marketing materials or improving our ability to provide technical support.

In general, testing is better if the assessment of quality is more accurate, the cost of testing is lower, the basis for making decisions is better, and the time frames are shorter. Perfect testing would be to instantly and effortlessly give the right information to allow any part of the business to make any necessary decision concerning the product.

By this definition, perfect testing is easy to achieve in some screwed-up projects. One example comes from my colleague Doug Hoffman, who was once in a situation where he was told that no testing he could do would affect the decision to ship. So he pronounced the testing complete. In a different situation, perhaps continuing the testing would have offered a benefit for technical support or provided a basis for some other type of corporate decision. The point is that when the testing is not coupled with a decision to be made and is not providing data for future use, that testing has no purpose.

Another all-too-common situation arises when an organization or regulatory authority requires certain testing or certain test products even if they contribute poorly or not at all to the project. Although I can see their political necessity, such test products have little to do with what Im proposing. Good Enough testing is about conscious and purposeful testing, not superstition and ritual. Most test plans that Ive seen could be torn up and thrown away with absolutely no effect on the test project or any stakeholders.

1. Assess product quality
How are we assessing and reporting the quality of the product?
Are we sure our assessment of quality is justified by our observations?
Are we aware of the stated and implied requirements of the product when we need to know them?
How quickly are we finding out about important problems in the product after they are created?
Are our tests covering the aspects of the product we need to cover?
Are we using a sufficient variety of test techniques or sources of information about quality to eliminate gaps in our test coverage?
What is the likelihood that the product could have important problems we dont know about?
What problems are reported through means other than our test process, that our testing should have found first?

2. Evaluate the cost of testing
How much does the testing cost? How much can we afford?
How can we eliminate unnecessary redundancy in our test coverage? What makes it difficult (and thus costly) to perform testing?
How might the product be made more testable?
Are there tools or techniques that might make the process more efficient or productive?
Would testing be less expensive overall if we had started sooner or waited until later?

3. Check how well testing
supports decision-making
Is the test process aware of the kinds of decisions management, developers, or other clients need to make?
Is the test process focused on potential product and project risks?
Is the test process tied to processes of change control and project management?
Are test reports delivered in a timely manner?
Are test reports communicated in a comprehensible format?
Is the test process communicated, as well as test results? Are we reporting the basis for our assessment and our confidence in it?
Is the test process serving the needs of technical support, publications, marketing, or any other business process that should use the quality assessment?

4. What about timing?
Every aspect of the three other parts of the model is time-driven. Thats the problem: We never have enough time to do everything, so everything we do is a race against the clock.

Full article...


Other Resource

... to read more articles, visit http://sqa.fyicenter.com/art/

A Framework for Good Enough Testing