Bug Advocacy
By: Cem Kane
Bug Advocacy
1. The point of testing is to find bugs.
2. Bug reports are your primary
work product. This is what people outside
of the testing group will most notice and most
remember of your work.
3. The best tester isn’t the one who finds the The best tester isn’t the one who finds the
most bugs or who embarrasses the most most bugs or who embarrasses the most
programmers. The best tester is the one programmers. The best tester is the one
who gets the most bugs fixed. who gets the most bugs fixed.
4. Programmers operate under time constraints and
competing priorities. For example, outside of the 8-
hour workday, some programmers prefer sleeping
and watching Star Wars to fixing bugs.
A bug report is a tool that you use to sell the
programmer on the idea of spending her time
and energy to fix a bug.
Note: When I say “the best tester is the one who gets the most bugs fixed,” I am not encouraging bug
counting metrics, which are almost always counterproductive. Instead, what I am suggesting is that
the effective tester looks to the effect of the bug report, and tries to write it in a way that gives each
bug its best chance of being fixed. Also, a bug report is successful if it enables an informed business
decision. Sometimes, the best decision is to not fix the bug. The excellent bug report raises the issue
and provides sufficient data for a good decision.
Selling Bugs Selling Bugs
Time is in short supply. If you want to
convince the programmer to spend her
time fixing your bug, you may have to sell
her on it.
(Your bug? How can it be your bug? The
programmer made it, not you, right? It’s
the programmer’s bug. Well, yes, but you
found it so now it’s yours too.)
Sales revolves around two fundamental
objectives:
• Motivate the buyer (Make her WANT to
fix the bug.)
• Overcome objections (Get past her
excuses and reasons for not fixing the
bug.)
Motivating the Bug Fixer
Some things that will often make
programmers want to fix the bug:
• It looks really bad.
• It will affect lots of people.
• Getting to it is trivially easy.
• It has embarrassed the company, or a bug
like it embarrassed a competitor.
• One of its cousins embarrassed the
company or a competitor.
• Management (that is, someone with
influence) has said that they really want it
fixed.
• You’ve said that you want the bug fixed,
and the programmer likes you, trusts your
judgment, is susceptible to flattery from
you, owes you a favor or accepted bribes
from you.
Overcoming Objections
Things that will make programmers resist
spending their time on the bug:
• The programmer can’t replicate the defect.
• Strange and complex set of steps required to
induce the failure.
• Not enough information to know what steps are
required, and it will take a lot of work to figure
them out.
• The programmer doesn’t understand the report.
• Unrealistic (e.g. “corner case”)
• It will take a lot of work to fix the defect.
• A fix will introduce too much risk into the code.
• No perceived customer impact
• Unimportant (no one will care if this is wrong:
minor error or unused feature.)
• Management doesn’t care about bugs like this.
• The programmer doesn’t like / trust you (or the
customer who is complaining about the bug).
Motivating Bug Fixes
By Better Researching
The Failure Conditions
Motivating The Bug Fix: Motivating The Bug Fix:
Looking At The Failure Looking At The Failure
Some vocabulary
• An error (or fault) is a design flaw or a
deviation from a desired or intended state.
• An error won’t yield a failure without the
conditions that trigger it. Example, if the
program yields 2+2=5 on the 10th time you
use it, you won’t see the error before or
after the 10th use.
• The failure is the program’s actual
incorrect or missing behavior under the
error-triggering conditions.
• Defect is frequently used to refer to the
failure or to the underlying error.
Motivating The Bug Fix: Motivating The Bug Fix:
Looking At The Failure
VOCABULARY EXAMPLE
Here’s a defective program
INPUT A
INPUT B
PRINT A/B
What is the fault?
What is the critical condition?
What will we see as the failure?
Motivating the Bug Fix
When you run a test and find a failure,
you’re looking at a symptom, not at the
underlying fault. You may or may not
have found the best example of a failure
that can be caused by the underlying
fault.
Therefore you should do some follow-up
work to try to prove that a defect:
>> is more serious than it first
appears.
>> is more general than it first
appears.
Motivating the Bug Fix: Motivating the Bug Fix:
Make it More Serious
LOOK FOR FOLLOW-UP ERRORS
When you find a coding error, you have the program
in a state that the programmer did not intend and
probably did not expect. There might also be data
with supposedly impossible values.
The program is now in a vulnerable state. Keep
testing it and you might find that the real impact of the
underlying fault is a much worse failure, such as a
system crash or corrupted data.
I do three types of follow-up testing:
• Vary my behavior (change the conditions by
changing what I do)
• Vary the options and settings of the program
(change the conditions by changing something
about the program under test).
• Vary the software and hardware environment.
Follow-Up: Vary Your Behavior
Keep using the program after you see the problem.
• Bring it to the failure case again (and again). If the program
fails when you do X, then do X many times. Is there a
cumulative impact?
• Try things that are related to the task that failed. For example,
if the program unexpectedly but slightly scrolls the display
when you add two numbers, try tests that affect adding or that
affect the numbers. Do X, see the scroll. Do Y then do X, see
the scroll. Do Z, then do X, see the scroll, etc. (If the scrolling
gets worse or better in one of these tests, follow that up, you’re
getting useful information for debugging.)
• Try things that are related to the failure. If the failure is
unexpected scrolling after adding, try scrolling first, thenadding.
Try repainting the screen, then adding. Try resizing the
display of the numbers, then adding.
• Try entering the numbers more quickly or changing the speed
of your activity in some other way.
• And try the usual exploratory testing techniques. So, for
example, you might try some interference tests. Stop the
program or pause it or swap it just as the program is failing. Or
try it while the program is doing a background save. Does that
cause data loss corruption along with this failure?
Follow-Up: Vary Options and Settings
In this case, the steps to achieve the failure are
taken as given. Try to reproduce the bug when
the program is in a different state:
• Use a different database.
• Change the values of persistent variables.
• Change how the program uses memory.
• Change anything that looks like it might be
relevant that allows you to change as an
option.
For example, suppose the program scrolls
unexpectedly when you add two numbers.Maybe
you can change the size of the program window,
or the precision (or displayed number of digits) of
the numbers, or background the activity of the
spell checker.
A bug might show a more serious failure if you run the
program with less memory, a higher resolution printer, with
more (or fewer) device interrupts coming in etc.
• If there is anything involving timing, use a really slow
computer, a really slow link, a really slow modem or printer.
And use very fast ones.
• If there is a video problem, try higher resolutions on the video
card. Try displaying MUCH more complex images (and much
simpler ones).
Note that we are not:
• checking standard configurations
• asking how broad the range of circumstances is that produces
the bug.
What we’re asking is whether there is a particular
configuration that will show the bug more spectacularly.
Returning to the example (unexpected scrolling when you
add two numbers), try things like:
• Different video resolutions
• Different mouse settings if you have a wheel mouse that does
semi-automated scrolling
• An NTSC (television) signal output instead of a traditional
(XGA or SVGA, etc.) monitor output.
IS THIS BUG NEW TO THIS VERSION?
In many projects, an old bug (from a previous shipping
release of the program) might not be taken very
seriously if there weren’t lots of customer complaints.
• (If you know it’s an old bug, check its complaint
history.)
• The bug will be taken more seriously if it is new.
• You can argue that it should be treated as new if
you can find a new variation or a new symptom that
didn’t exist in the previous release. What you are
showing is that the new version’s code interacts
with this error in new ways. That’s a new problem.
• This type of follow-up testing is especially
important during a maintenance release that is just
getting rid of a few bugs. Bugs won’t be fixed
unless they were (a) scheduled to be fixed because
they are critical or (b) new side effects of the new
bugfixing code.
Motivating the Bug Fix:
Show it is More General Show it is More General
LOOK FOR CONFIGURATION DEPENDENCE
Bugs that don’t fail on the programmer’s machine are much less credible
(to that programmer). If they are configuration dependent, the report will
be much more credible if it identifies the configuration dependence
directly (and so the programmer starts out with the expectation that it
won’t fail on all machines.)
In the ideal case (standard in many companies), you test on 2 machines
• Do your main testing on Machine 1. Maybe this is your powerhouse:
latest processor, newest updates to the operating system, fancy
printer, video card, USB devices, huge hard disk, lots of RAM, cable
modem, etc.
• When you find a defect, use Machine 1 as your bug reporting
machine and replicate on Machine 2. Machine 2 is totally different.
Different processor, different keyboard and keyboard driver,
different video, barely enough RAM, slow, small hard drive, dial-up
connection with a link that makes turtles look fast.
• Some people do their main testing on the turtle and use the power
machine for replication.
• Write the steps, one by one, on the bug form at Machine 1. As you
write them, try them on Machine 2. If you get the same failure,
you’ve checked yo ur bug report while you wrote it. (A valuable
thing to do.)
• If you don’t get the same failure, you have a configuration
dependent bug. Time to do troubleshooting. But at least you know
that you have to.
AS A MATTER OF GENERAL GOOD PRACTICE, IT PAYS TO
REPLICATE EVERY BUG ON A SECOND MACHINE.
TRY VARIANTS THAT SHOULDN’T MATTER
The point of this exercise is to show that you get
the same failure whether a given variable is set
one way or another.
In follow-up testing, we varied “irrelevant”
variables with an eye to seeing differences in the
failure symptoms. We picked variables that
looked promising for this.
In generalization testing, we’re still looking to see
whether the failure symptoms change, but we’re
picking variables that we don’t expect to cause a
change. The point is to take a variable that has
been set one way throughout the testing of this
bug, show that you get the same problem with a
different setting, and then you can ignore this
variable, not discuss it in the bug report, treat it
as irrelevant.
UNCORNER YOUR CORNER CASES
We test at extreme values because these are the most
likely places to show a defect. But once we find the
defect, we don’t have to stick with extreme value tests.
• Try mainstream values. These are easy settings
that should pose no problem to the program. Do
you replicate the bug? If yes, write it up, referring
primarily to these mainstream settings. This will be
a very credible bug report.
• If the mainstream values don’t yield failure, but the
extremes do, then do some troubleshooting around
the extremes. Is the bug tied to a single setting (a
true corner case)? Or is there a small range of
cases? What? In your report, identify the narrow
range that yields failures. The range might be so
narrow that the bug gets deferred. That might be
the right decision. In some companies, the product
has several hundred open bugs a few weeks before
shipping. They have to decide which 300 to fix
(the rest will be deferred). Your reports help the
company choose the right 300 bugs to fix, and help
people size the risks associated with the remaining
ones.
Overcoming Objections
Via Analysis of the Failure
Things that will make programmers
resist spending their time on the bug:
•The programmer
can’t replicate the
defect.
• Strange and complex set of steps required
to induce the failure.
• Not enough information to know what
steps are required, and it will take a lot of
work to figure them out.
• The programmer doesn’t understand the
report.
• Unrealistic (e.g. “corner case”)
Objection, Objection: Objection, Objection:
Non Non-Reproducible Errors
Always report non-reproducible errors. If you report
them well, programmers can often figure out the
underlying problem.
To help them, you must describe the failure as
precisely as possible. If you can identify a display or a
message well enough, the programmer can often
identify a specific point in the code that the failure had
to pass through.
• When you realize that you can’t reproduce the bug,
write down everything you can remember. Do it now,
before you forget even more. As you write, ask yourself
whether you’re sure that you did this step (or saw this
thing) exactly as you are describing it. If not, say so.
Draw these distinctions right away. The longer you wait,
the more you’ll forget.
• Maybe the failure was a delayed reaction to something
you did before starting this test or series of tests. Before
you forget, note the tasks you did before running this
test.
• Check the bug tracking system. Are there similar
failures? Maybe you can find a pattern.
• Find ways to affect timing of your program or of your
devices, Slow down, speed up.
• Talk to the programmer and/or read the code.
Non Non-Reproducible Errors
• The fact that a bug is not reproducible is data. The
program is telling you that you have a hole in your
logic. You are not entertaining certain relevant
conditions. Why not?
• See Watts Humphrey, Personal Software Process, for
recommendations to programmers of a system for
discovering and then eliminating characteristic errors
from their code. A non-reproducible bug is a tester’s
error, just like a design bug is a programmer’s error.
It’s valuable to develop a system for discovering your
blind spots. To improve over time, keep track of the
bugs you’re missing and what conditions you are not
attending to (or find too hard to manipulate).
• The following pages give a list of some conditions
commonly ignored or missed by testers. Your personal
list will be different in some ways, but maybe this is a
good start. When you run into a irreproducible defect
look at this list and ask whether any of these conditions
could be the critical one. If it could, vary your tests on
that basis and you might reproduce the failure.
Other Resource
... to read more articles, visit http://sqa.fyicenter.com/art/
|