Calculate averages
most commonly used, and most commonly misunderstood, statistic of all. To calculate an
average, you simply add up all the numbers and divide the sum by the quantity of
numbers you just added. What seems to confound many people the most when it comes
to performance testing is that, in this example, Data Sets A, B, and C each have an
average of exactly 4. In terms of application response times, these sets of data have
extremely different meanings. Given a response time goal of 5 seconds, looking at only
the average of these sets, all three seem to meet the goal. Looking at the data, however,
shows that none of the data sets is composed only of data that meets the goal, and that
Data Set B probably demonstrates some kind of performance anomaly. Use caution when
using averages to discuss response times and, if at all possible, avoid using averages as
the only reported statistic. When reporting averages, it is a good idea to include the
sample size, minimum value, maximum value, and standard deviation for the data set.
is a straightforward concept that is easier to demonstrate than define. For example, to find
the 95
down six data points from the largest. The 6
"95 percent of the simulated users experienced a response time of [the 6
It is important to note that percentile statistics can only stand alone when used to
represent data that is uniformly or normally distributed with an acceptable number of
outliers (see "Statistical Outliers" below). To illustrate this point, consider the exemplar
data sets. The 95
misleading as well because the 80
16 seconds, so in this case neither of these percentile values represent the data in a
manner that is useful to summarizing response time.