Software QA FYI - SQAFYI

Uncharted Territory: Introducing QA in a Web Startup

By: Lisa Crispin

Much has been written, in this newsletter and other publications, about the risks of e-Business applications. "Web-time" is a widely acknowledged phenomenon. We all agree that quality is imperative for an e-Business, as all the competition is just a click away. Unfortunately, most of us can agree that a Web startup is not an environment in which quality testing is typically found.

Development is fast and loose. Many developers are inexperienced. Marketing is pushing to be beat the competition to market. The rules change every day. An e-Business needs to respond immediately to market pressures. And an e-Business cannot afford poor performance or that big revenue drain, downtime.

I have now survived bringing quality assurance (QA) and testing to two startups - one an e-Business,, the other a software development shop, Tensegrent. By sharing my experiences, I will provide guidelines for successful implementation of QA and testing in the startup environment.

When I accepted the opportunity of being the first test engineer at, the startup consisted of 25 employees. The developers had produced some exciting applications and felt they were ready to "grow up and play with the big boys." The development team thought they were intellectually prepared to introduce standards and procedures.

In reality, development was frenetic, and the developers didn't have a clue as to how to stop and analyze their processes, much less how to impose discipline on them.

What's worse, the wide wild world of the Web was completely new to me. I truly felt lost in the wilderness without a map. My background was in testing traditional software in mid- to large-sized software companies. I had worked on every possible platform and operating system, tested databases and client/server software, but knew nothing about Internet applications. I knew my customers' needs. I was used to development cycles of several months - during which your typical Web application has gone through numerous incarnations.

Using the strategy I will outline below, I overcame these limitations. I collaborated with the software and product developers and marketing managers to implement development standards and project processes that build quality into the applications. now employs over two hundred people and has over four million registered customers. It was acquired in March, 2000 for $326 million by Galileo International Inc.

Recently I moved on to Tensegrent, getting in much earlier in the lifecycle of a startup - I was hired in the second month of the company's existence, the 10th employee and the first Test Engineer. Tensegrent is devoted to developing high-quality software through the use of the eXtreme Programming (XP) technique. The good news was that XP involves intense unit and integration testing on the part of the developers. The bad news was that the role of the Test Engineer was not well-defined.

Any startup or DotCom is a work in progress. Even once we had a successful project process at, we faced continual challenges such as an inadequate test environment. Even when the whole company understands and is committed to the importance of quality assurance testing, unexpected events lead to surprises. The key is to keep plugging away at the following tasks:

* Get Buy-In
* Work Smart
* Define Processes

I. Get Buy-In
I won over my managers, developers, and marketing counterparts by following these tenets: Identify a top manager in your organization who believes in your cause and will champion it. At, it was the Vice President of Web Development and Chief Cat Herder (yes, that really was her title). When I was hired, this person did not believe that five developers could keep one tester busy full time. But, in time, she became my biggest ally. She not only pushed the developers to work for quality, but she also lobbied the management team for testing resources. At Tensegrent, the staff was actively committed to quality, but I needed help from the General Manager to get them on track.

Partner with someone outside of your organization, such as a project or operations manager. Educate your ally to garner his or her help. At, we had a topnotch project manager who, once she understood what QA and testing would do for the company, did much of my job for me. She enforced processes such as document review and signoff, helped implement and police the defect tracking system, and tied up a million details involved with every big production launch. She became the prime channel of communication between marketing and development. At Tensegrent, our team was the whole company, at the beginning; the team captains were key to enforcing process.

Educate everyone you come into contact with at the company about software quality assurance: what it is and what it will do for them and the products. Early on at, I held a "Lunch 'n' learn" professional development seminar. The company bought lunch, so attendance was good. I explained why testing is essential and what it involves. At Tensegrent, I held a similar Brown Bag session where I demonstrated the automated test tool to the developers. Lots of meetings and one-on-one encounters are needed to get everyone on board and to establish priorities.

Support Information Systems, the group that administers the production site. These people (or person, if your company is really small) will benefit from not having their pager go off so much when applications are tested before being launched to production. The Vice President of Technology and the IS director at fully supported me and refused to launch any update that had not been fully tested. This kept the rest of the organization from steamrolling over me, giving me a chance to prove the value of testing. I don't bug IS for the things I can do myself, but I let them do their job. For example, at, I installed Y2K patches on the test machines, but IS controlled all the UNIX and NT account management.

Understand the developers' point of view. You may have young, brilliant developers who don't communicate in ways you are accustomed to.'s original developers were mostly very young and inexperienced. Some had not finished high school!. Others weren't old enough to drink! The culture was anti-corporate, and they said what was on their minds. I found that if I listened, I learned, and they in turn were willing to listen to my ideas. I learned everything I now know about Web applications from the developers themselves.

Celebrate success. At, I presented a "Quality Hero" award each month to a developer who took exceptional measures to prevent defects and improve quality. The prize was just a Nerf gun and the developer's name engraved on a plaque, but it raised the visibility of high quality process and techniques. At Tensegrent, I try to make sure the celebratory end-of-iteration Foosball games occur AFTER acceptance testing is successfully completed!

Brainstorm with developers and others about problems that may not come out in testing. For example, I didn't know that if you change a URL, search engines may not be able to find your site. When we implemented a content management tool at that required changing every URL, this was important information!

II. Work Smart
Here's my advice for making the testing organization lean and mean. This is especially critical in an Extreme Programming environment or anywhere the ratio of developers to testers is high. Evaluate tools. Put as much time as you can into tool evaluation, such as those for automated testing, defect tracking, and configuration management. Identify the vendors who can help you the most, and get as much information from them as you can. Ask fellow testers for their recommendations and experiences. Install new tools and try them out. Select tools that are appropriate for you and your company. It doesn't do any good to buy a tool you don't have time to learn how to use, especially if your testing team is small. I ended up choosing tools that are lesser known but still meet our needs. For example:

* For automated testing, both and Tensegrent use WebART (, an inexpensive, easy-to-learn, but powerful tool sold by OCLC Inc. (a non-profit company). The vendor gave me valuable advice and provided insights about Web testing. Pick everyone's brain, including vendors!
* For defect tracking at, we chose a Web-based tool, TeamTrack (now called TTrack), from a startup company called TeamShare . It, also was far less expensive than its competitors, but it was easy to implement and customize. At Tensegrent, which is a brand-new startup on a small budget, we use the free Bugzilla successfully.
* For configuration management, at we again turned to a smaller, innovative company which produces an inexpensive, easy to implement and learn yet robust tool, Perforce. At Tensegrent, we use freeware, CVS - it lacks some features, but the development team is small and can work around its drawbacks more easily.
* These tools won't necessarily meet your needs - just be open and creative when evaluating tools. Investigate alternatives - for example, if you create unit tests to reproduce each bug you find in testing, you may not even need a defect tracking system!

Design automated tests that work for you.

Automated acceptance tests at Tensegrent and use the following criteria to provide useful tests with minimum resources:
* Modular and self-verifying to keep up with the pace of development in "Web-time".
* Verify the minimum criteria for success. Make sure the developers write comprehensive unit tests. Acceptance tests can't cover every path through the code.
* Perform each function in one and only one place to minimize maintenance time
. * Contain modules that can be reused, even for unrelated projects
* Do the simplest thing that works. This XP value applies as much to testing as to coding.
* Report results in easy-to-read format. Create easy-to-read reports from your test results and post them so that everyone can monitor status and keep the project on track. The project team will gain confidence as they see the percentage of successful tests increases.

In addition, the developers try to design the software with testability in mind. This might mean building hooks into the application to help automate acceptance tests. Push as much functionality as possible to the backend, because it is much easier to automate tests against a backend than through a user interface.

Search the Web for resources. I would have quit after a week if I had not found an excellent Web site that points to information about testing Web applications and lists of tools. These sites, in turn, led me to more tools and information Here are some examples:

Full article...

Other Resource

... to read more articles, visit

Uncharted Territory: Introducing QA in a Web Startup