When Should a Test Be Automated?
By: Brian Marick
I want to automate as many tests as I can. Iím not comfortable running a test only once.
What if a programmer then changes the code and introduces a bug? What if I donít catch
that bug because I didnít rerun the test after the change? Wouldnít I feel horrible?
Well, yes, but Iím not paid to feel comfortable rather than horrible. Iím paid to be costeffective.
It took me a long time, but I finally realized that I was over-automating, that
only some of the tests I created should be automated. Some of the tests I was automating
not only did not find bugs when they were rerun, they had no significant prospect of doing
so. Automating them was not a rational decision.
The question, then, is how to make a rational decision. When I take a job as a contract
tester, I typically design a series of tests for some product feature. For each of them, I
need to decide whether that particular test should be automated. This paper describes
how I think about the tradeoffs.
In order for my argument to be clear, I must avoid trying to describe all possible testing
scenarios at once. You as a reader are better served if I pick one realistic and useful
scenario, describe it well, and then leave you to apply the argument to your specific
situation. Hereís my scenario:
1. You have a fixed level of automation support. That is, automation tools are available.
You know how to use them, though you may not be an expert. Support libraries have
been written. I assume youíll work with what youíve got, not decide to acquire new
tools, add more than simple features to a tool support library, or learn more about test
automation. The question is: given what you have now, is automating this test
justified? The decision about what to provide you was made earlier, and you live with
In other scenarios, you might argue for increased automation support later in the
project. This paper does not directly address when thatís a good argument, but it
provides context by detailing what it means to reduce the cost or increase the value of
2. There are only two possibilities: a completely automated test that can run entirely
unattended, and a "one-shot" manual test that is run once and then thrown away.
These are extremes on a continuum. You might have tests that automate only
... to read more articles, visit http://sqa.fyicenter.com/art/