By: Mitch Allen
How do you test a Web site anyway? That's the first thing I asked myself when I became the SQA Manager for a dot-com. I knew
how to build Web sites, and I had tested several commercial products. It would be easy right? Nope. Not only is it hard, but
it can also be expensive.
Performance and Scalability
If you are managing Web site
testing for an application that will be used by a large number of people, your number one concern will be performance
testing. When several people start using the Web site at the same time, performance may degrade, database transactions might
overlap, etc. Your job is to make sure that that doesn't happen.
The first thing you should ask in an interview
for Web testing is "will it scale?" Now what does that mean? You want to work on a Web site that is "n-tier." That means that it has
at least three layers (tiers). The layer that the user of the site sees is the GUI (Graphical User Interface) layer (also
referred to as the front-end or presentation layer). Behind that, in the middle is the middle tier. This is where all the
business logic happens. It's also referred to as the application layer. This is where the application servers go. Behind that is the
database tier also referred to as the back-end.
In an n-tier architecture, the GUI layer should be concerned with
nothing but rendering the Web page and passing transaction requests to the middle tier. No heavy duty calculations should be
performed in the user's browser. It should happen on the Web site in the application layer.
The GUI layer must
never, ever, ever talk to the database directly. (Side bar: I'd like to point out that I violate these rules everyday - but
I'm not developing large commercial n-tier sites).
The middle tier is the heart of the application and the key to
scalability. When a user makes a transaction, like filling out a form on the GUI layer, the middle tier is where the
transaction is handled. The middle tier will talk to the database and do some of the transaction processing. In J2EE architecture, this layer is written using Enterprise JavaBeans. In
.NET systems, this layer may be written in a language like Visual
The database (back-end) can also play an important role in transaction processing. The first step in improving
a Web site's performance can be to move as much of the middle tier logic as possible to stored procedures in the database.
For a Web site to scale, it must be determined how many users a middle tier server can handle. This is done through performance testing. If the middle tier is
designed properly, adding another server will double the amount of users that the system can handle. If you can serve up 1
transaction per second for 500 users using 1 middle tier server, then 3 servers should be able to maintain that rate for 1500
users (500 x 3). Though realistically there will always be an acceptable amount of degradation as you add a new server.
No Web site can handle an infinite amount of users. Performance requirements have to be realistic. You may have heard
in the news about Denial of Service (DoS) attacks. This is basically performance testing run amok. Hackers hijack a bunch of
servers to act as agents and flood a server with transactions until they overwhelm the system. Instead of breaking the
system, your job is to determine how many users it can handle before it breaks.
To test site scalability and performance, you need a testing product that can emulate x number of users
hitting your site. Tools to do this range from being free, to $100,000 depending on how many "virtual users" you want to
emulate. I could go on about how ridiculous it is to charge an extra $20,000 just to up an internal counter, but I'll
refrain. Just be careful when shopping for tools.
Here is a list of some performance tools:
Segue Silk Performer
Mercury Interactive LoadRunner
Be forewarned that some of the more
expensive tools have very restrictive licensing software. I was amazed to find out that after spending a fortune on one
product that only one of my employees at a time could run the editor to write tests! They expected me to spend another
$10,000 to get a special meter to allow multiple editors (I declined).
You will also need to invest in at least
one machine that will act as an agent, emulating hits against your test servers. I've used as many as three. Your vendor can
tell you what you will need.
Before you launch into a testing effort,
you should find out if there is enough money in the budget to emulate the production environment. You don't want to do tests
against the live site.
You need to setup a "staging" environment which emulates your production environment on a
smaller scale. You will need a router, just like the one in production (or if money is really tight, something as close to it
as possible), at least two Web servers and two middle tier servers and one database server. And don't forget the firewalls.
If the hardware doesn't match the production environment, then testing on the staging servers won't help you find problems in
the production environment. You need to work closely with the production team to make sure that your test environment
emulates every bit of hardware and software in the production environment.
Your staging servers need to exist with
your performance testing environment in an isolated network. Usually a patch panel is setup so you can pull one plug and
isolate a rack full of servers at will. When doing performance testing, all numbers will be useless if your environment is
suffering from noise not only from the corporate net, but the Internet as well.
And lock them down! Make sure developers
don't have ftp, write, or PC Anywhere access to your servers. Otherwise unseen changes to your staging environment could
wreak havoc on your test results.
Routers, Firewalls and SSL
Large Web sites don't
have just one Web server. When you visit the site, a router will redirect you to one of many servers. The job of the router is to balance the load by trying
to evenly distribute users amongst the many servers. In order for the system to work properly, all Web servers need to be
exact copies of one another. So that users can get to the GUI layer servers, they need to be outside the company firewall.
Even if you are just working
on a Web application destined for internal use, users may still want to use it from outside the firewall. For example, your
sales force may need access from a client site to give a demo or access the application. Regardless, you must test using a
firewall. Firewalls go between the GUI layer and middle tier and it's not uncommon to have a firewall between the middle tier
and the back-end. You will find that an application that worked fine in an open environment collapses once a firewall is
introduced. Firewalls act as choke points which could cut off or interfere with the communication between tiers.
Secure Socket Layer
(SSL) is another issue that you need to deal with. If you've ever entered your credit card number online and noticed a little
lock symbol in your browser, that's SSL working. Another sign is when the Web address changes from "http" to "https." Private
SSL servers require a complex procedure to install a special certificate for access. Users of a commercial site should not
have to go through such a procedure to create a secure connection. You need to verify that external users can access the
secure layer without a problem.
Cookies and Session Management
The Web is a stateless
environment. This means that a lot of data isn't stored on the client side. When you visit a site and "talk" to a Web server,
it doesn't remember who you are. Though you can help it remember if you let it set a "cookie" in your browser. This is how sites remember your name, etc. When you visit the site again it asks
your browser for the cookie, which may contain something like your login ID.
Cookies are good for long term
memory, but for short term memory, Web applications use session management. Session management is sort of like a cookie, but it will expire after a certain
amount of inactivity (like 30 minutes). Think of it like a Session ID.
As you surf a site, you may end up talking
to several servers at once. Each transaction may be dished out to a different server. As you move from server to server, your
session information needs to stay with you or the next server will have no idea what you are talking about. For example, when
you get to the checkout server, session information will help it remember what items you ordered. For testing, you need to
make sure that as a user moves through the system that session information is carried with them and hasn't expired in a
reasonable amount of time.
Components and Configuration Management
When trying to
setup bug tracking you may come to a horrifying conclusion. The concept of a "build number" for the overall product may not
work. Web site development can be very fluid. People will update script, image, and source files through FTP or file copying
without even thinking about. The concept of "builds" may even be foreign to some Web developers. You need to work with
development to determine what defines a build when entering bugs. Ideally your Web site will be broken up into components
that can be tested in isolation and combined and rolled out as sets.
When testing, you want to be able to certify
components and component sets. For example, you may certify that Contact Form 1.1 works well with Order App 1.3, but not 1.2.
Because of the fluid nature of the Web, the site probably won't be updated all at once. Developers will usually want to "drop
in" a component to the live site or a client's site. That's why configuration management and certification are so important.
Before Component X 2.3 is dropped in with Component Y 2.4, you need to certify that everything will still work.
your site uses ActiveX or some other technology
to install components on the browser side, you need to deal with that as well. If your application uses browser side objects,
Testing Under the Hood
The GUI layer often talks to the middle tier by tossing back and forth
XML files. The trend is moving towards hiding the XML files as Web
Service calls. You should ask your development team if the Web Services are published in the form of an internal or
external document. If you have a programming background, you could emulate the GUI layers communication with the middle tier
by automating Web Service calls. If you are comfortable with SQL, you could talk to development about unit testing stored
procedures in the database layer.
Testing an enterprise level Web site can
be an overwhelming and expensive task. Before embarking on such an effort, it's important to make sure that upper management
will commit the time, money, and resources required. Given the proper budget, schedule, knowledge, and commitment from
Development, you should be able to handle the task.
Mitch Allen has been working in SQA since 1989,
specializing in test automation. For more information please visit his Web site at http://www.mitchallen.com.
Article © 2003 Mitch Allen