Software QA FYI - SQAFYI

Testing embedded systems: Do you have the GuTs for it?

By: Vincent Encontre

This article from The Rational Edge offers a general introduction to testing embedded systems, including a discussion of how embedded systems issues affect testing process and technologies, and how Rational Test RealTime provides solutions for these issues.

This article offers a general introduction to testing embedded systems, including a discussion of how embedded systems issues affect testing process and technologies, and how Rational Test RealTime provides solutions for these issues. Many thanks to Paul Szymkowiak, Rational's process engineer involved in the testing portion of the Rational Unified Process® (RUP®), for his improvements to my original text, for his continuous support, and for his glossary of real-time embedded terminology at the conclusion of this article.

In order to settle on a common set of concepts, let's start with some definitions.

What is testing? Testing is a disciplined process that consists of evaluating the application (including its components) behavior, performance, and robustness -- usually against expected criteria. One of the main criteria, although usually implicit, is to be as defect-free as possible. Expected behavior, performance, and robustness should therefore be both formally described and measurable. Verification and Validation (V&V) activities focus on both the quality of the software product and of the engineering process. These V&V activities can be sub-classified as either preventative, detective, or corrective measures of quality. While testing is most often regarded as a detective measure of quality, it is closely related to corrective measures such as debugging. In practice, software developers usually find it more productive to enact testing and debugging together, usually as an interactive process. Debugging literally means removing defects ("bugs").

What exactly is an "embedded system"? It's difficult, and highly controversial, to give a precise definition. So, here are some examples. Embedded systems are in every "intelligent" device that is infiltrating our daily lives: the cell phone in your pocket, and all the wireless infrastructure behind it; the Palm Pilot on your desk; the Internet router your e-mails are channeled through; your big-screen home theater system; the air traffic control station as well as the delayed aircraft it is monitoring! Software now makes up 90 percent of the value of these devices.

Most, if not all, embedded systems are "real-time" (the terms "real-time" and "embedded" are often used interchangeably). A real-time system is one in which the correctness of a computation not only depends on its logical correctness, but also on the time at which the result is produced. If the timing constraints of the system are not met, system failure is said to have occurred. And for some systems, identified as safety-critical, failure is not an option.

Thus testing timing constraints is as important as testing functional behavior for an embedded system.

Even though the embedded systems testing process resembles that used for many other kinds of applications, it is impacted by some issues important to the embedded world:
* Separation between the application development and execution platforms
* A large variety of execution platforms and thus of cross-development environments
* A wide range of deployment architectures
* Coexistence of various implementation paradigms
* Tight resources and timing constraints on the execution platform
* Lack of clear design models
* Emerging quality and certification standards

A more detailed discussion of these issues is found near the end of this article.

These issues greatly affect testability and measurability of an embedded system. This explains why testing such systems is so difficult and consequently is one of the weakest points of current development practices. So it is no wonder that, according to a recent study: 50 percent plus of embedded systems development projects are months behind schedule and only 44 percent of designs are within 20 percent of feature and performance expectations, and all this even when 50 percent plus of total development effort is spent in testing.

In the rest of this article, we will:
* Go through a generic test iteration from which we will derive a minimal set of desired testing technology.
* Instantiate this iteration to address the testing of complex systems as found in the embedded world; based on these considerations we will add capabilities to our ideal testing technology.
* Moving one step further, examine what makes embedded systems so difficult to develop and test, and assess how these issues add to the list of features that need to be fulfilled by the technology used to test them. Rational Test RealTime will be used as an example of a product that implements a large portion of this ideal technology.

A Generic Test Iteration

The first step we will consider is to identify the granule to be tested. I use the word granule to avoid other words such as component, unit, or system, all of which have less generic definitions. For example, in the Unified Modeling Language (UML) a component is a piece of an application that has its own thread of control (that is, a task or a Unix process). I also like the word granule as the root for granularity, which translates well over the wide range of elements that can be tested: from a single line of code up to a large distributed system.

Preparing the Granule Under Test
So, first identify the granule to test, then transform it to a testable granule, or granule under test (GuT). This step consists of isolating the granule from its environment by making it independent with the help of stubs and, sometimes, adapters. A stub is a piece of code that simulates two-way access between the granule and another part of the application. Next, build a test driver. This will stimulate the GuT with the appropriate input, measure output information, and compare it with the expected response. An adapter is sometimes used to allow the GuT to be stimulated by the test driver. Stimulation and measurement follow specific paths through gates in the GuT: we will call them Points of Control and Observation (PCOs), a term that comes directly from the telecommunications industry. A PCO can be located at the border of, or inside, the GuT. Examples of PCOs for a C function granule are:

* Point of Observation inside the granule: coverage of a specific line of code in the function.
* Point of Observation at the border of the granule: parameter values returned by the function.
* Point of Control inside the granule: change of a local variable.
* Point of Control at the border of the granule: the function call with actual parameters.

Stubs and even test drivers can be other parts of the application (if they are available) and don't necessarily need to be developed for testing a C function or a C++ class. The GuT can be accessed or stimulated through another part of the application, which then plays the role of stub or test driver. Stubs and test drivers constitute the test harness environment of the GuT.

Defining the Test Case

Defining a test case is a matter of figuring out:
* The appropriate PCOs: this depends on the kind of testing to be performed, such as functional, structural, load, and so on
* How to exploit these PCOs: which information to send and to expect to receive through them, and in what order, if any

The type of test as well as sent/expected information is driven by the requirement set for the GuT. In the case of safety-critical systems, formal and precise requirements are an essential part of the development process. While formal requirements are an important motivation source for tests, they don't always explicitly identify the tests that will discover important flaws in the system. Using the formal requirements, and for less critical applications when specific requirements are lacking, the tester must consider an appropriate set of tests to conduct (also referred to as "test ideas") to identify some requirements for testing the GuT. These requirements can then be translated into concrete test cases that take advantage of available PCOs. By "concrete," we primarily mean executable.

Usually, requirements themselves are not formal and do not naturally translate into formal test cases. This translation process often introduces errors in which test cases do not accurately reflect requirements. Specification languages have become more formal since the introduction of UML, and it has now become possible to express formal requirement-based test cases to avoid the translation pitfall. Rational QualityArchitect and a part of Rational Test RealTime provide good examples of using such model-based testing techniques.

Unfortunately, not all requirements are described using UML: in the embedded world, the most common formal description technique for a test case is simply to use a programming language such as C or C++. While C or C++ are universally known (which reduces the learning curve), they don't do a good job of taking into account test case needs such as PCO definition or expected GuT behavior. As such they make it difficult for someone to define and implement comprehensive, formal test cases. This problem has been addressed by the design of specific high-level testing languages, which are well adapted to specific testing domains such as data-intensive or transaction-based testing. Rational Test RealTime proposes a mix of native 3GL and dedicated high-level scripting languages, bringing the best of both worlds: reduced learning curve and efficiency in defining and implementing test cases.

Another extremely valuable and productive way to implement one or more test cases is to use session recorders: While the GuT is stimulated (either manually or by its future environment), particular Points of Observation record information passing in and out of the GuT. This information is automatically post-processed and translated into an appropriate case to be replayed later. An example of such a session recorder is found in Rational Rose RealTime where model execution leads to the generation of a UML sequence diagram reflecting trace execution, which is then used as input to Rational QualityArchitect RealTime.

Each test implementation must bring the GuT to a particular starting state that allows the test to be run. This part of a test is known as the preamble . Symmetrically, at the end of the effective test and whatever the result, the test implementation must bring the GuT to a final stable state that allows the next test to execute. This part of the test is called the postamble .

Deploying and Executing the Test

Once implemented, the test case is then integrated with the test driver and stubs. It is important to note that stub definition and implementation is an integral part of the test, enabling the test case to be realized. The test case forms the informational (vs. the operational) part of the test, and is validated during test harness execution.

Observing Test Results

Results from test execution are monitored through Points of Observation.

At the border of the granule, typical types of Points of Observation include:

* Parameters returned by functions or received messages.
* Value of global variables.
* Ordering and timing of information.

Inside the granule, typical types of Points of Observation include:

* Source code coverage, providing details about which software part of the GuT has been covered.
* Information flow to visualize exchange of information with respect to time between the different parts of the GuT. Typically this kind of flow is represented as a UML sequence diagram as in Rational Test RealTime.
* Resource usage showing nonfunctional information such as time spent in the various parts of the GuT, memory pool management (as exemplified in the Rational PurifyPlus family), or event handling performances.

All these observations can be collected for a single test case and/or aggregated for a set of test cases.

Full article...

Other Resource

... to read more articles, visit

Testing embedded systems: Do you have the GuTs for it?