Software QA FYI - SQAFYI

How to Quantify Quality: Finding Scales of Measure

By: Tom Gilb

‘Scales of measure’ are fundamental to a specification method we have developed called Planguage. They are central to the definition of all scalar attributes; that is, to all the performance (especially quality attributes) and resource attributes.

You can learn the art of developing your own tailored scales of measure for the performance and resource attributes, which are important to your organization or system. You cannot rely on being 'given the answer' about how to quantify. You will lose control over your current vital system performance concerns if you cannot or do not quantify the critical attributes.

Finding and Developing Scales of Measure and Meters

The basic advice for identifying and developing scales of measure and meters (practical methods for measuring) for scalar attributes is as follows:
1. Try to re-use previously defined Scales and Meters. Examples [Posem. www].
2. Try to modify previously defined Scales and Meters.
3. If no existing Scale or Meter can be reused or modified, use common sense to develop innovative home-grown quantification ideas.
4. Whatever Scale or Meter you start off with, you must be prepared to learn. Obtain and use early feedback, from colleagues and from field tests, to redefine and improve your Scales and Meters.

Reference Library for Scales of Measure

‘Reuse’ is an important concept for sharing experience and saving time when developing Scales. You need to build reference libraries of your ‘standard’ scales of measure. Remember to maintain details supporting each ‘standard’ Scale, such as Source, Owner, Status and Version (Date). If the name of a Scale’s designer is also kept, you can probably contact them for assistance and ideas. Here is a template for keeping reusable scales of measure.

Tag: <assign a tag name to this Scale>.
Version: <date of the latest version or change>.
Owner: <role/email of who is responsible for updates/changes>.
Status: <Draft, SQC Exited, Approved>.
Scale: <specify the Scale with defined [qualifiers].>
Alternative Scales: <reference by tag or define other Scales of interest as alternatives and supplements>.
Qualifier Definitions: <define the scale qualifiers, like ‘for defined [Staff]’, list their options, like {Nurse, Doctor, Orderly}.>.
Meter Options: <suggest Meter(s) appropriate to the Scale>.
Known Usage: <reference projects & specifications where this Scale was actually used in practice with designers’ names>.
Known Problems: <list known or perceived problems with this Scale>.
Limitations: <list known or perceived limitations with this Scale>.

Example: This is a draft template, with , for specification of scales of measure in a reference library. Many of the terms used here are defined in Competitive Engineering [www & CE]. See example below for sample use of this template.

Tag: Ease of Access.
Version: 11-Aug-2003.
Owner: Rating Model Project (Bill).
Scale: Speed for a defined [Employee Type] with defined [Experience] to get a defined [Client Type] operating successfully from the moment of a decision to use the application.

Alternative Scales: None known yet.
Qualifier Definitions:
* Employee Type: {Credit Analyst, Investment Banker, …}.
* Experience: {Never, Occasional, Frequent, Recent}.
* Client Type: {Major, Frequent, Minor, Infrequent}.
Meter Options:
* Test all frequent combinations of qualifiers at least twice. Measure speed for the combinations.

Known Usage: Project Capital Investment Proposals [2001, London].
Known Problems: None recorded yet.
Limitations: None recorded yet.

Example of a ‘Scale’ specification for a Scale reference library. This exploits the template in the previous example.

Reference Library for Meters

Another important standards library to maintain is a library of ‘Meters.’ Meters support scales of measure by providing practical methods for actually measuring the numeric Scale values. ‘Off the shelf’ Meters from standard reference libraries can save time and effort since they are already developed and are more or less ‘tried and tested’ in the field.

It is natural to reference suggested Meters within definitions of specific scales of measure (as in the template and example above). Scales and Meters belong intimately together.

Managing ‘What’ You Measure
It is a well-known paradigm that you can manage what you can measure. If you want to achieve something in practice, then quantification, and later measurement, are essential first steps for making sure you get it. If you do not make critical performance attributes measurable, then it is likely to be less motivating for people to find ways to deliver necessary performance levels. They have no clear targets to work towards, and there are no precise criteria for judgement of failure or success.

Practical Example: Scale Definition

‘User-friendly’ is a popular term. Can you specify a scale of measure for it?

Here is my advice on how to tackle developing a definition for this.

1. If we assume there is no ‘off-the-shelf’ definition that could be used (there are [POSEM, CE]):
* Be more specific about the various aspects of the quality. There are many distinct dimensions of qualities such as usability, maintainability, security, adaptability and their like [CE]. List about 5 to 15 aspects of some selected quality that is critical to your project.
* For this example, let’s select ‘environmentally friendly’ as the one of many aspects that we are interested in, and we shall work on this below as an example.

2. Invent and specify a Tag: ‘Environmentally Friendly’ is sufficiently descriptive. Ideally, it could be shorter, but it is very descriptive left as it is. We indicate a ‘formally defined concept’ by capitalizing the tag. Tag: Environmentally Friendly.
Note, we usually don’t explicitly specify ‘Tag: ’ but this sometimes makes the tag identity clearer.

3. Check there is an Ambition statement, which briefly describes the level of requirement ambition. ‘Ambition’ is a defined Planguage parameter. More parameters follow, below.
Ambition: A high degree of protection, compared to competitors, over the short-term and the long-term, in near and remote environments for health and safety of living things.

4. Ensure there is general agreement by all the involved parties with the Ambition definition. If not, ask for suggestions for modifications or additions to it. Here is a simple improvement to my initial Ambition statement. It actually introduces a ‘constraint’.
Ambition: A high degree of protection, compared to competitors, over the short-term and the long-term, in near and remote environments for health and safety of living things, which does not reduce the protection already present in nature.

5. Using the Ambition description, define an initial ‘Scale’ (of measure) that is somehow quantifiable (meaning – you can meaningfully attach a number to it). Consider ‘what will be sensed by the stakeholders’ if the level of quality changes. What would be a ‘visible effect’ if the quality improved? My initial, unfinished attempt, at finding a suitable ‘Scale’ captured the ideas of change occurring, and of things getting ‘better or worse’:
Scale: The % change in positive (good environment) or negative directions for defined [Environmental Changes].
My first Scale parameter draft, with a single scalar variable.
However, I was not happy with it, so I made a second attempt. I refined the Scale by expanding it to include the ideas of specific things being effected in specific places over given times:
Scale: % destruction or reduction of defined [Thing] in defined [Place] during a defined [Time Period] as caused by defined [Environmental Changes].

This is the second Scalar definition draft with four scalar variables. These will be more-specifically defined whenever the Scale is applied in requirement statements such as ‘Goal’.

This felt better. In practice, I have added more [qualifiers] into the Scale, to indicate the variables that must be defined by specific things, places and time periods whenever the Scale is used.

6. Determine if the term needs to be defined with several different scales of measure, or whether one like this, with general parameters, will do. Has the Ambition been adequately captured? To determine what’s best, you should list some of the possible sub-components of the term (that is, what can it be broken down into, in detail?). For example:
Thing: {Air, Water, Plant, Animal}.
Place: {Personal, Home, Community, Planet}.
Thing: = {Air, Water, Plant, Animal}.
Place: Consists of {Personal, Home, Community, Planet}.

Definition examples of the scale qualifiers used in the examples above. The first example means: ‘Thing’ is defined as the set of things Air, Water, Plan and Animal (which, since they are all four capitalized, are themselves defined elsewhere). Instead of just the colon after the tag, the more explicit Planguage parameter ‘Consists Of’ or ‘=’ can be used to make this notation more immediately intelligible to novices in reading Planguage.

Then consider whether your defined Scale enables the performance levels for these sub-components to be expressed. You may have overlooked an opportunity, and may want to add one or more qualifiers to that Scale. For example, we could potentially add the scale qualifiers ‘…. under defined [Environmental Conditions] in defined [Countries]…’ to make the scale definition even more explicit and more general.

Scale qualifiers (like …‘defined [Place]’…) have the following advantages:
* they add clarity to the specifications
* they make the Scales themselves more reusable in other projects
* they make the Scale more useful in this project: specific benchmarks, targets and constraints can be specified for any interesting combination of scale variables (such as, ‘Thing = Air’).

7. Start working on a ‘Meter’ – a specification of how we intend to test or measure the performance of a real system with respect to the defined Scale. Remember, you should first check there is not a standard or company reference library Meter that you could use. Try to imagine a practical way to measure things along the Scale, or at least sketch one out. My example is only an initial rough sketch.

Meter: {scientific data where available, opinion surveys, admitted intuitive guesses}.

This Meter specification is a sketch defined by a {set} of three rough measurement concepts. These at least suggest something about the quality and costs with such a measuring process. The ‘Meter’ must always explicitly address a particular ‘Scale’ specification.
The Meter will help confirm your choice of Scale as it will provide evidence that practical measurements can feasibly be obtained on a given Scale of measure.

8. Now try out the Scale specification by trying to use it for specifying some useful levels on the scale. Define some reference points from the past (Benchmarks) and some future requirements (Targets and Constraints). For example:

Full article...

Other Resource

... to read more articles, visit

How to Quantify Quality: Finding Scales of Measure