Software QA FYI - SQAFYI

Do Not Get Out of Control: Achieving Real-time Quality and Performance


When lives are at risk if systems fail, it is critical to minimize defects through the best software engineering processes possible. High-maturity processes are valuable for delivering quality, mission-critical software and supporting overall project performance.

Control Chart Evolution: A Case Example
This section describes the evolution of the control charts used in our organization to develop life- and mission-critical software in a project-based environment. The evolution of our charts is driven by the data that we collect. With more data, we discover different, and often better, ways to utilize the data. Our organization started looking at the “what and how” of measuring process performance back in 2004. We did this by defining organization-wide project metrics. Activities addressed include planning, system requirements, software requirements, software design, software coding, etc. These are based on regulatory requirements, such as DO-178B, “Software Considerations in Airborne Systems and Equipment Certification” [4], which regulates avionics software. Our team identified key quality and performance measures that could easily be captured and would provide meaningful information.

the code development activity, it is easy to track lines of source code, number of requirements implemented, code development time, code review time, number of defects, rework time (fixing defects), and rework verification time (time spent ensuring the defects were fixed properly). We track key activities utilizing some existing tools within our organization, with minimal burden on engineers or managers. Today there are COTS tools many organizations can use to do this. Next came the development of organizational baseline control charts. Before claiming an organization baseline, our team requires two criteria be satisfied. First, there must be enough data points to ensure the baseline is statistically stable—a minimum of 40. Second, there must be a mix of enough different projects to generalize the findings to other projects. This is more of a qualitative decision based on knowledge of the projects.

After generating the first baselines, we noticed very high UCLs, indicating a great deal of variance. With a high UCL, some projects never generated out of control points, whereas other projects consistently triggered run rules. This was the result of developing a mixture of complex systems.

The team looked into the types of complex projects involved,such as avionics and medical systems. Organizational baselines were then split by application area. The resulting medical control charts became more useful at detecting outliers, but the avionics control charts were still not performing to the level desired. The DO-178B guideline for avionics classifies features based on the criticality of failure from Level A to E. Failure of a Level A feature would result in a catastrophic failure condition for an aircraft. Level A design, development, testing and defects are treated much different than Level E failure, which are associated with defects that will not have an effect on aircraft operational capability or pilot workload .

Initially, these levels appeared to be a great way to partition the work. However, the problem partitioning baseline control charts by DO-178B is that a project may have requirements at multiple levels, so it is too difficult to accurately track all measures to these levels.

The team also looked at whether the project dealt predominantly with embedded or non-embedded software. This partitioning could be applied once at the very beginning of a project, simplifying the identification of the project type. Embedded avionics and non-embedded avionics organization control charts had very different CLs and UCLs. The variation from the control limits to the CL was statistically significant for the two control charts. But, we were still not done.

Within the embedded/non-embedded control charts, some reviews had much lower defect rates than others. The major differences were based on what was being reviewed. Higher defect rates were associated with new code that was reviewed for the very first time. Lower defect rates were associated with existing code that had been reworked and were undergoing a subsequent review due to additional code change.

Certain projects differed based on customer and type of work, so customer- and job-specific control chart baselines were created. Some of these differences are based on a customer’s process maturity levels. A customer’s process maturity level can impact how well they define system and software-related requirements. It also influences how stable the system and software requirements or target hardware remain during a project.

Even with stable organizational baselines the team strives to reexamine the CLs and control limits every six months. This ensures the process improves over time, since as a service-based organization our customer and project mix is changing over time, and with new technologies.

Hypothesis Testing
Evidence is gathered when considering whether to partition an organizational control chart. Some evidence is based on knowledge and experience with our business. Our team also uses statistical tools to identify and study potential differences.

When an organizational control chart is updated, part of the process is to look for differences among customers and projects. When differences are identified, they are statistically analyzed using t-tests or analysis of variance to determine whether the differences are actually statistically significant.

When customer or project data are statistically different the project management team determines why the difference exists. This drives the decision as to whether a more specific organizaWithin tional control chart should be created. A new organizational chart is warranted if we expect to be doing similar projects in the future. One-time projects that are unlikely to be repeated are excluded from the organizational control charts. The information is documented just in case projects similar to this do start popping up.

Full article...

Other Resource

... to read more articles, visit

Do Not Get Out of Control: Achieving Real-time Quality and Performance