Software QA FYI - SQAFYI

What is a Serious Bug?

By: Cem Kaner

How Should We Define a Serious Defect?
Failure to conform to specifications is a common theme in legal books, but many of the software development contracts provide vague, incomplete specifications that will change over time without being updated in the contract itself. Discussions within the software development community consistently recognize that most failures in commercial software products are due to errors in the specifications or requirements. A widely used number is that 80% of the money spent fixing or dealing with software problems can be traced back to requirements errors.

As a result, texts that focus on software errors don't limit themselves to failure to meet a specification (this type of failure is called nonconformance). Here are some examples from well respected texts in the field:

IEEE ( 1989), IEEE Standard Dictionary of Measures to Produce Reliable Software, ANSI/IEEE Standard 982.1-1988, p. 13:

# Defect: A product anomaly. Examples include such things as (1) omissions and imperfections found during early life cycle phases and (2) faults contained in software sufficiently mature for test or operation. See also fault.

IEEE (1994), IEEE Standard Classification for Software Anomalies, IEEE Standard 1044-1993, p. 3.

# Anomaly: Any condition that deviates from expectations based on requirements specifications, design documents, user documents, standards, etc., or from someone's perceptions or experiences. Anomalies may be found during, but not limited to, the review, test, analysis, compilation, or use of software products or applicable documentation.

# A defect is any flaw in the specification, design, or implementation of a product. . . . If a flaw could not possibly have been detected, or if it could have been detected and would not have been corrected then it is an enhancement. Defects do not include typographical or grammatical errors in engineering documentation.

Explains why he doesn't trust quality as measured in terms of compliance with standards and specifications. The problem is that these are not true measures of the quality of the product. He works through an excellent example dealing with a role of newsprint. The true measure of quality is whether the paper rips while on the rotary press. Published standards, in terms of such things as tensile strength, provide only secondary measures of how the product will perform in the field. Continuing, on page 56, "There are no standards-whether they be national, international, or company-wide-that are perfect. Usually standards contain some inherent defects. Consumer requirements also change continuously, demanding higher quality year after year. Standards that were adequate when they were first established, quickly become obsolete. []We engage in QC to satisfy customer requirements"

A software defect is simply a bug which if not removed would cause a program or system to fail or to produce incorrect results. Note: the very common idea that a defect is a failure to adhere to some user requirements is unsatisfactory because it offers no way to measure requirements defects themselves, which constitute one of the larger categories of software error.

Any variation from the specifications is a nonconformity . . . . There is a group of nonconformities which represent serious threats to the welfare of users and bystanders. These nonconformities are called defects, and they not only can cause injury but may also result in the manufacturers, designers, or sellers being sued under the product liability laws. There are also a class of defects called design defects which can be responsible for customer dissatisfaction, loss, injury or death. Despite the fact that all of the product conforms to the design, the product is faulty and is not properly designed.

One common definition is that a software occurs when the software does not perform according to its specifications. This definition has one fundamental flaw: it tacitly assumes that the specifications are correct. This is rarely, if ever, a valid assumption: one of the major sources of errors is the writing of specifications. . . . [] A second common definition is that an error occurs when the software does not perform according to its specifications providing that it is used within its design limits. This definition is actually poorer than the first one. . . . . [] A third possible definition is that an error occurs when the software does not behave according to the official documentation or publications supplied to the user. Unfortunately, this definition also has several flaws. There exists the possibility that the software does behave according to the official publications but errors are present because both the software and the publications are in error. A second problem occurs because of the tendency of user publications to describe only the manual for a time-sharing system that states, "To enter a new command press the attention key once and type the command." Suppose that a user presses the attention key twice by accident and the software system fails because its designers did not plan for this condition. The system obviously contains an error, but we cannot really state that the system is not behaving according to its publications. [] The last definition that is sometimes used defines an error as a failure of the software to perform according to the original contract or documentation. Although this definition is an improvement over the previous three, it still has several flaws . . . written user requirements are rarely detailed enough to describe the desired behavior of the software under all possible circumstances. [] There is, however, a reasonable definition of a software error that solves the aforementioned problems: A software error is present when the software does not do what the user reasonably expects it to do."

# "Software defects can be divided into four broad categories: (1) requirements defects, (2) design defects, (3) code defects, and (4) documentation defects."

Whether or not the specification or user documentation addresses the following issues, all of them should be considered material bugs, shouldn't they?

* the product doesn't work at all

* the disks are blank

* the product caused substantial harm to the property or the business of the licensee

* the licensor supplied a product that lack promised or advertised features or capabilities

* the software deprives the customer of a key benefit that she reasonably expected (for example, anyone would expect a word processor to be able to print and it's unreasonable if a particular one can't provide that benefit)

* the customer spent so much time, effort, and money dealing with a bug that the customer's costs exceeded the cost of the product. (If you think that this isn't a serious enough loss to the customer, what if the customer's losses run at ten times the cost of the software, or one hundred times the cost? At some point, the amount of the losses caused by the software becomes excessive, doesn't it?)

Full article...

Other Resource

... to read more articles, visit

What is a Serious Bug?