Software QA FYI - SQAFYI

Top 10 Tips on How to Improve a Game Quality Assurance Department

By: Phillip DeRosa

I'm director of quality assurance at Alberta, Canada-based BioWare Corp. BioWare is the developer of upcoming titles like the martial arts action-RPG Jade Empire and the fantasy RPG Dragon Age, among others. I've been meaning to write a piece like this one for a while now - hopefully, some of this information will prove useful to some of you regarding practical tips for improving quality assurance regarding approaches to video game bug and standard testing.

1. Define "Quality" and "Fun"
So, what is "quality" anyway? Some of the common definitions are: "a degree of excellence" or "a distinguishing attribute," which aren't very useful to those of us who have to assure that it's there.
Let's begin by defining quality within the context of the products we make (note: defining quality will be a common theme in this article).
Games are complex pieces of software. We can start by breaking them down into smaller, more manageable chunks. For example we could start by looking at the software side; we can devise test approaches to our games by applying many of the standard techniques from the software quality and assurance industry. These "best practices" are worth examining in more detail: A few examples could be: functional testing (does it work as intended?), performance testing (does it meet our performance targets?) and usability (ease of use) to name just a few.
Now comes the fun partů literally! For such a small word, "fun" is a vast topic. Actually, I'd go so far as to call it a science. This means applying an approach of observation, identification, description, experimental investigation, and theoretical explanation to help define the seemingly simple word, "fun". As we did with "quality", let's define "fun" by asking some simple questions. This process will lead to some answers, and probably even more questions. A few examples could be: why do people play (our) games? What is fun? What isn't fun? Note that the specific questions you ask may well vary depending on the kinds of games you develop.

2. Make a Plan and Set a Vision
Planning is one of the four pillars of management. We should all have plans, right? It is said that "when you fail to plan, you plan to fail." This will also serve as a roadmap for our departments. Planning ability could be used as a measure of our progress as leads (scary isn't it).
The vision is simply over-arching goals we hope to achieve with the plan. Our current QA department's vision is to develop a team of workers whose goal is to improve product quality while reducing the cost of bugs by finding them earlier.

3. Provide your team with clear Job Descriptions and Responsibilities
As a profession QA can provide a challenging and rewarding experience, and for some it can lead to a lateral movement to other departments. We are often required to be jacks-of-all-trades and masters of at least one. Testing requires an understanding of a lot of different systems and their interactions as well as the ability to communicate our ideas to the developers.
We have a variety of roles within QA. Some are dictated by corporate goals, other are dictated by the types of games we make and by the type of information we seek. For example, our full-time testers provide expert testing services and developer support. Currently our full-time roles include QA leads, technical and design gurus (sub-leads), QA tools leads, technical testers, gameplay/story testers, application testers and QA programmers.
Our contract testing roles include the proverbial playtesters, focus testers, beta testers and user test specialists. Let's talk about the playtesters (who by the way are an essential part of the QA team, albeit at a later stage of development): they are energetic and provide fresh perspectives. It also helps me sleep better at night knowing that a small army of playtesters are taking the boot to our games.
I've been mulling over for a while now how regression testing works with regard to our games. Regression testing is simply ensuring changes introduced do not adversely affect current functionality. It's near impossible to perform full regression on games such as the ones made at BioWare. By using automation and build acceptance tests we can cover what we need to. At a certain point we add a test case which requires playing through the entire game's critical path. This is a good task for a playtester.

4. Find Good People, and Treat them Well! - Quality in the Workplace
Another of BioWare's core values (along with "Quality in our Products") is "Quality in our Workplace". Basically, we believe that a company's greatest asset is its people. A good place to start when creating a high performance team is to surround yourself with talented and passionate individuals.
Let's look at the hiring process. Hiring is serious business which has a direct and high impact on team performance. What is it that carpenters say? - "measure twice, cut once." When we bring someone aboard we are confident they will be a valuable addition to our team. (The idea of spending the majority of one's time coaching low performance employees or remaining in hiring hell isn't appealing nor productive).
Retaining great people is just as vital for any company. Investing time in finding out what motivates (and de-motivates, so you can improve things!) is just as important. A good manager listens attentively; remember that good ideas can come from anywhere and anyone, anytime. Career planning is not just nice benefit but essential, and in some cases succession planning may be required.

5. Training and Development
The products we create are the results of our collective intellects and a goodly sum of our blood, sweat and tears. Training is one conduit by which we can hone our knowledge, not to mention that it's mutually beneficial to both employee and employer. Employees share and apply this new found knowledge to the products they help to create. Employees feel valued and appreciated when their employers invest in their training and long term career development - this helps to build long term loyalty to the company.
Discussion can be a powerful ally. This prompted us to develop a few programs with that very idea in mind. Our Design Group focuses on gameplay and design discussions. Our Reading Group looks at writing and story elements. The QA critique session is where the team watches gameplay in the BioWare movie theatre and openly discuss their thoughts. We are in the planning stages of a programming discussion group as well. Additionally, we maintain a library of books covering multiple disciplines such as game theory, quality assurance, leadership and programming.
Professional development should also be considered. I'd describe these as universal skills - the type of skills we can apply at work or in our everyday lives. A few examples would be decision making, time management and communication.

6. Tools of the Trade
Labor is by far the biggest expense. That's why employee productivity should be at the top of our list of priorities. Tools have been around for eons; it's a simple concept really: tools help us work more efficiently and achieve better results. With that said, are we providing our teams the tools to make them better workers?
The problem with tools is that they usually have a high initial cost or the benefits aren't immediately obvious. I take it we're all familiar with lost productivity (this seemingly innocuous spectre of inefficiency). It doesn't show up on balance sheets, and to borrow a words of Verbal Kint from the Usual Suspects, "the greatest trick the Devil ever pulled was convincing the world that he didn't exist." Seek and ye shall find! Go straight to the source. Talk to the employees and find out what frustrates them. Observe them in their natural habitat. The project management tool can be a powerful ally, if the probes are set right. We can apply our experience and understanding of the process to identify the real problem. Once we gathered this information all we need to do is pull it all into a Return on Investment (ROI) or similar report and present it to management to help back up the case for improving the tools.
Tools can also help eliminate repetitive and monotonous tasks. Bill Gates summed it up well when he said, "it's hard to measure when you make your knowledge workers more effective, but it's just common sense that knowledge workers who are not distracted by or burdened by routine matters will do better work." (Bill Gates, Business @ The Speed of Thought)
In our department the popular tools are project management, debugging and logging, development, automation and benchmarking tools. In addition we've created a game research station where we can perform competitive analysis.

Full article...

Other Resource

... to read more articles, visit

Top 10 Tips on How to Improve a Game Quality Assurance Department