background image
<< Report early and report often | Use Concise Verbal Summaries >>
<< Report early and report often | Use Concise Verbal Summaries >>

Summarize Data Effectively

do not enjoy statistics. This can lead to significant misrepresentations of performance test
results when reporting. If you are not sure what statistics to use to highlight a particular
issue, do not hesitate to ask for assistance.
Consolidate Data Correctly
While it is not strictly necessary to consolidate results, it tends to be much easier to
demonstrate patterns in results when those results are consolidated into one or two graphs
rather than distributed over dozens. That said, it is important to remember that only
results from identical test executions that are statistically similar can be consolidated into
performance report output tables and charts.
Additional Considerations
In order for results to be consolidated, both the test and the test environment must be
identical, and the test results must be statistically equivalent. One approach to
determining if results are similar enough to be consolidated is to compare results from at
least five test executions and apply the following rules:
·
If more than 20 percent (or one out of five) of the test execution results appear not to
be similar to the rest, something is generally wrong with the test environment, the
application, or the test itself.
·
If a 95
th
percentile value for any test execution is greater than the maximum or less
than the minimum value for any of the other test executions, it is not statistically
similar.
·
If every page/timer result in a test execution is noticeably higher or lower on the chart
than the results of all the rest of the test executions, it is not statistically similar.
·
If a single page/timer result in a test execution is noticeably higher or lower on the
chart than all the rest of the test execution results, but the results for all the rest of the
pages/timers in that test execution are not, the test executions are probably
statistically similar.
Summarize Data Effectively
Summarizing results frequently makes it much easier to demonstrate meaningful patterns
in the test results. Summary charts and tables present data from different test executions
side by side so that trends and patterns are easy to identify. The overall point of these
tables and charts is to show team members how the test results compare to the
performance goals of the system so they can make important decisions about taking the
system live, upgrading the system, or even, in some cases, completely reevaluating the
project.
Additional Considerations
Keep the following key points in mind when summarizing test data:
·
Use charts and tables that make your findings clear.
·
Use text to supplement tables and charts, not the other way around.
·
If a chart or table is confusing to the reader, don't use it.