<< Baseline and benchmark | Archive tests >>

Benchmarking

For example, you could measure load time, number of transactions processed per unit of
time, number of Web pages served per unit of time, and resource utilization such as
memory and processor usage. Some considerations about using baselines include:
·

A baseline can be created for a system, component, or application.

A baseline can be created at different layers: database, Web services, etc.

A baseline can be used as a standard for comparison to track future optimizations or
regressions. When using a baseline for this purpose, it is important to validate that the
baseline tests and results are well understood and repeatable.

Baselines can help product teams articulate variances that represent degradation or
optimization during the course of the development life cycle by providing a known
starting point for trend analysis. Baselines are most valuable if created using a set of
reusable test assets; it is important that such tests are representative of workload
characteristics that are both repeatable and provide an appropriately accurate
simulation.

Baseline results can be articulated by using combinations of a broad set of key
performance indicators such as response time, processor, memory, disk, and network.

Sharing baseline results across the team establishes a common foundation of
information about performance characteristics to enable future communication about
performance changes in an application or component.

A baseline is specific to an application and is most useful for comparing performance
across different builds, versions, or releases.

Establishing a baseline before making configuration changes almost always saves
time because it enables you to quickly determine what effect the changes had on the
application's performance.

Benchmarking

Benchmarking is the process of comparing your system performance against an industry
standard that is endorsed by some other organization.

From the perspective of Web application development, benchmarking involves running a
set of tests that comply with the specifications of an industry benchmark to capture the
performance metrics for your application necessary to determine its benchmark score.
You can then compare your application against other systems or applications that have
also calculated their score for the same benchmark. You may choose to tune your
application performance to achieve or surpass a certain benchmark score. Some
considerations about benchmarking include:

A benchmark score is achieved by working within industry specifications or by
porting an existing implementation to comply with those specifications.

Benchmarking generally requires identifying all of the necessary components that
will run together, the market where the product exists, and the specific metrics to
measure.

Benchmark scores can be published publicly and may result in comparisons being
made by competitors.

Performance metrics that may be included along with

benchmark scores include response time, transactions processed per unit of time, Web
pages accessed per unit of time, processor usage, memory usage, and search times.