A Framework for Good Enough Testing
By: James Bach
GOOD ENOUGH TESTING DEFINED
In any situation, Good Enough testing
asks, “How do I know if I’m doing, or
have done, enough of the right type of
testing?” Unfortunately, there is no
objective or rigorous calculus for answering
this question, but we can identify
what to consider in attempting to answer
it. We can begin to build at least a heuristic
framework around the problem. In
fact, my general model of Good Enough
quality could apply to Good Enough testing
as well. I’m borrowing some elements
from it, but here I’m proposing a more
specific framework.
As a first step, I’ll define the term:
Good Enough testing is the process of
developing a sufficient assessment of
quality, at a reasonable cost, to enable
wise and timely decisions to be made
concerning the product.
Let me unpack this definition. I’m
asserting that, at a high level, it’s useful to
think about the value of testing as a
dynamic with four parts:
• Assessment of product quality (How
accurate and complete is it?).
• Cost of testing (How reasonable is
it? Is it within project constraints?
Is there a good return on investment,
such as the extent of information
gained per test?).
• Decisions (How well does the
assessment serve the project and the
business?).
• Timing of all the above (Is it soon
enough to be useful?).
Testing also supports business processes
beyond decision-making. But for this
short column, I am lumping under “decisions”
the uses any testing client would
have for test results, such as creating accurate
marketing materials or improving our
ability to provide technical support.
In general, testing is better if the assessment
of quality is more accurate, the cost
of testing is lower, the basis for making
decisions is better, and the time frames are
shorter. Perfect testing would be to
instantly and effortlessly give the right
information to allow any part of the business
to make any necessary decision concerning
the product.
By this definition, perfect testing is easy
to achieve in some screwed-up projects.
One example comes from my colleague
Doug Hoffman, who was once in a situation
where he was told that no testing he
could do would affect the decision to ship.
So he pronounced the testing complete.
In a different situation, perhaps continuing
the testing would have offered a
benefit for technical support or provided
a basis for some other type of corporate
decision. The point is that when the testing
is not coupled with a decision to be
made and is not providing data for future
use, that testing has no purpose.
Another all-too-common situation
arises when an organization or regulatory
authority requires certain testing or
certain test products even if they contribute
poorly or not at all to the project.
Although I can see their political necessity,
such test products have little to do
with what I’m proposing. Good Enough
testing is about conscious and purposeful
testing, not superstition and ritual. Most
test plans that I’ve seen could be torn up
and thrown away with absolutely no
effect on the test project or any stakeholders.
1. Assess product quality
• How are we assessing and reporting
the quality of the product?
• Are we sure our assessment of quality
is justified by our observations?
• Are we aware of the stated and
implied requirements of the product
when we need to know them?
• How quickly are we finding out
about important problems in the
product after they are created?
• Are our tests covering the aspects of
the product we need to cover?
• Are we using a sufficient variety of
test techniques or sources of information
about quality to eliminate
gaps in our test coverage?
• What is the likelihood that the product
could have important problems
we don’t know about?
• What problems are reported through
means other than our test process,
that our testing should have found
first?
2. Evaluate the cost of testing
• How much does the testing cost?
How much can we afford?
• How can we eliminate unnecessary
redundancy in our test coverage?
• What makes it difficult (and thus
costly) to perform testing?
• How might the product be made
more testable?
• Are there tools or techniques that
might make the process more efficient
or productive?
• Would testing be less expensive
overall if we had started sooner or
waited until later?
3. Check how well testing
supports decision-making
• Is the test process aware of the kinds
of decisions management, developers,
or other clients need to make?
• Is the test process focused on potential
product and project risks?
• Is the test process tied to processes
of change control and project management?
• Are test reports delivered in a timely
manner?
• Are test reports communicated in a
comprehensible format?
• Is the test process communicated, as
well as test results? Are we reporting
the basis for our assessment and
our confidence in it?
• Is the test process serving the needs
of technical support, publications,
marketing, or any other business
process that should use the quality
assessment?
4. What about timing?
Every aspect of the three other parts of
the model is time-driven. That’s the problem:
We never have enough time to do
everything, so everything we do is a race
against the clock.
Full article...
Other Resource
... to read more articles, visit http://sqa.fyicenter.com/art/
|