Software QA FYI - SQAFYI

When Should a Test Be Automated?

By: Brian Marick

I want to automate as many tests as I can. I’m not comfortable running a test only once. What if a programmer then changes the code and introduces a bug? What if I don’t catch that bug because I didn’t rerun the test after the change? Wouldn’t I feel horrible?

Well, yes, but I’m not paid to feel comfortable rather than horrible. I’m paid to be costeffective. It took me a long time, but I finally realized that I was over-automating, that only some of the tests I created should be automated. Some of the tests I was automating not only did not find bugs when they were rerun, they had no significant prospect of doing so. Automating them was not a rational decision.

The question, then, is how to make a rational decision. When I take a job as a contract tester, I typically design a series of tests for some product feature. For each of them, I need to decide whether that particular test should be automated. This paper describes how I think about the tradeoffs.

Scenarios

In order for my argument to be clear, I must avoid trying to describe all possible testing scenarios at once. You as a reader are better served if I pick one realistic and useful scenario, describe it well, and then leave you to apply the argument to your specific situation. Here’s my scenario:

1. You have a fixed level of automation support. That is, automation tools are available. You know how to use them, though you may not be an expert. Support libraries have been written. I assume you’ll work with what you’ve got, not decide to acquire new tools, add more than simple features to a tool support library, or learn more about test automation. The question is: given what you have now, is automating this test justified? The decision about what to provide you was made earlier, and you live with it.

In other scenarios, you might argue for increased automation support later in the project. This paper does not directly address when that’s a good argument, but it provides context by detailing what it means to reduce the cost or increase the value of automation.

2. There are only two possibilities: a completely automated test that can run entirely unattended, and a "one-shot" manual test that is run once and then thrown away. These are extremes on a continuum. You might have tests that automate only

Full article...


Other Resource

... to read more articles, visit http://sqa.fyicenter.com/art/

When Should a Test Be Automated?