Software QA FYI - SQAFYI

Lightsabers, Time Machines, & Other Automation Heuristics Plus a Few Anti-heuristics

By: Adam Goucher

Heuristics are used in testing as rules of thumb or prompts for solving a particular problem or class of problems. An important characteristic of heuristics is that they are fallible in certain situations, though correct in most situations. And then there are anti-heuristics. They are fallible in most situations.

Automation, like all of testing, is an inherently heuristic activity. Some of the risks associated with it can be managed by making conscious decisions about which heuristics to employ and which anti-heuristics to avoid.

This article identifies a handful of heuristics that apply to automation. While not an exhaustive set, it is a useful one, and it will put you on the path to identifying and collecting your own set of automation heuristics.

In the Star Wars stories, the lightsaber is the physical tool the Jedi turn to when their primary tool, the Force, proves insufficient for the task at hand. In testing, we use automation when our primary tool, our brain, is insufficient for the task.

An important step in the apprenticeship of a young Jedi is the creation of their first lightsaber. This happens towards the end of their apprenticeship. Prior to that they use either a blunted lightsaber or one that is not attuned to their own energies.

This lack of attenuation is exactly what happens when using “generic” automation tools. It is not the vendor’s intent to be malicious or to do harm, but by their very nature these tools need to support a lot of situations to be viable in the market. That support leads to generic approaches, and that sometimes means you have to adapt the way your application is built to accomodate your automation tool.

When using a closed-source tool the challenges are even greater, as you are dependent on the vendor for any fixes to problems you might discover. And what is a problem in your context might not be one to them, so they might be unwilling to fix what they don’t see as a problem.

Just as a Jedi solves the problem of attenuation and bug fixing by building their own lightsaber, building your own automation tool or framework using Open Source components addresses these problems for your automation. No longer shackled to generic solutions to specific problems, you can use one that solves your specific problem precisely. And if some difficulty occurs, you have access to the source code and can produce a fix to it immediately.

The Lightsaber heuristic is of course fallible. A Padawan (apprentice Jedi) does not just wander out on their own and build a Lightsaber; they have a Master who guides them through the process the first time—in part to protect them from themselves. So too with building automation. Someone who does not have experience with a number of different automation tools or techniques could stumble upon something that works, but is more likely to head down false paths, run into dead ends, and end up causing more harm than good. My own current automation framework, Saunter is the 5th or 6th incarnation of these ideas and is only now at the point where it is really useful.

With Open Source, as with commercial automation, you still need to be aware of the “Know Where The Sun Is” heuristic when fixing problems you encounter. For example, the Selenium WebDriver API is Open Source and anyone can build custom copies, but if you have a question on the Java, Ruby, or Python implementations, the best way to get information is during European business hours. For C#, though, it is business hours on the east coast of the US. During WinRunner’s heyday if you tried to call in the (really) early morning North American time you hit the support center in Israel.

There will be times when technology choices dictate going with a commercial, closed source tool or framework. But teams that really succeed with automation tend to be ones that build their own tools using Open Source components.

One final thing to remember with lightsabers is that they are wielded for both good and evil purposes.

Time Machines
It is impossible to determine the Return on Investment (ROI) for automation without access to a time traveling device. This is a fairly outrageous claim to make in a world where budget justifications need to be made quarterly, but it really strikes at the core of why we use automation in the first place.

Automation is not a technique to be used to increase some metric in a spreadsheet tallying the number of scripts executed or coverage percentage. It’s not really even a technique for doing things faster than humans or that humans are not good at (like boring data input). Automation is just a technique for gathering information. And that information is consumed by humans who make decisions with it.

It is this “usage in decisions” that is the tricky bit. Consider the following situations:

Full article...

Other Resource

... to read more articles, visit

Lightsabers, Time Machines, & Other Automation Heuristics Plus a Few Anti-heuristics