Quantify Captured Performance Requirements
quantified and testable. If you are a little less lucky, the requirements you captured are
not quantified at all, in which case you can follow the process described above for
quantifying performance goals. If you are unlucky, the performance requirements that
you collected are partly quantified and non-testable.
The challenge is that if a requirement is extracted from a contract or existing marketing
document, it likely cannot be changed. When you are faced with a requirement such as
"three-second average response time," or "2,500 concurrent users," you have to figure
out what those requirements mean and what additional information you need in order to
make them testable.
There is no absolute formula for this. The basic idea is to interpret the requirements
precisely written in common language, supplement them with the most common or
expected state for the application, and then get your extended, testable requirement
approved by the stakeholder(s). The stakeholders will then be held responsible if
someone were to challenge legal compliance with the requirements after the product goes
live. To illustrate, consider the following examples:
Requirement: Direct quote from a legal contract: "The Website shall exhibit an average
response time of not greater than three (3) seconds."
Extended quantification: This requirement is particularly challenging. The literal, and
therefore most likely legal, interpretation is that "Over the life of the Website, the
arithmetic mean of all response times, at any point in time, will not exceed 3 seconds."
While that is hard enough to determine, response time has not been defined either.
Response time could mean "end-user-perceived response time," "server response time,"
or something else entirely. The following breaks this down systematically:
·
reasonable way to test the three-second average response time is either "with all
pages being accessed equally often" or "under the most likely workload distribution."
conditions for the test. In this case, your best bet is probably to average across
multiple volumes. For instance, you could get 30 percent of your data from low-load
tests, 50 percent from expected-load tests, and 20 percent from high-load tests, and
then report a weighted average -- assuming that distribution of load is a reasonable
approximation of the anticipated production load profile. Alternatively, you could
make a case for testing this requirement exclusively under expected load conditions.
Requirement: Direct quote from sales brochure: "This application will support up to
2,500 concurrent users."
Extended quantification: The challenge here is similar because "concurrent user" is not
technically accurate for Web applications and therefore can mean several different things.