Software QA FYI - SQAFYI

Testing maturity assessment

By: Luc Sauliere

To assess the maturity of an organization or a team on testing is often a starting point for our customers to implement improvement plans. The implementation and monitoring of these plans rely then on indicators, to ensure progress and results.

Our test expert partners are frequently asked to achieve these audits, develop plans, and implement indicators to follow. We worked to build dashboards that can contribute to this audit phase but also be a support for the monitoring of actions and showing the results.

These charts are based on the information generated for each project. They provide several ways to show this information with indicators:

o test coverage on overall application or only the modifications
o the number of iterations/delivery needed for validation
o the distribution of the testing effort between different types of tests, whether manual or automated, performed by the development team or the functional testers
o and other indocators as timing, modification rate or coverage evolution…

Portfolio Cartography
The Portfolio is available on the cockpit via the “Dashboard” menu, Mapping is available in the “ Evolution ” part of the Portfolio (see left menu).

A major way to show the information is based on a 2-dimensional graph. Its goal is to compare the testing activities of several projects, to highlight risky ones and to identify improvement areas.

Kalistick testing quadrants
Each quadrant has a specific meaning and you can choose what kind of information you want to display: information about the project or about the latest target version. The size of each disk is the size of the project or the latest version if you want to focus on the changes scope. The color represents the test coverage either on the overall application or only modifications.

Quadrant 1 Weakly tested
This area includes projects with a lack of test. Not only there is little testing in the development phase but also bit of functional tests. If further analysis confirms a quality problem, several improvement actions can be taken in the short and medium term.

Quadrant 2 Especially unit and integration tests

In this area, there are projects with development teams having reached a good maturity on testing. It usually means testing in a continuous integration process. There are more often agile teams, they choose to automate their tests and to do less manual functional test. There are also applications that mainly offer services (SOA) and therefore does not provide a user interface. However, there is little functional tests with end-user vision or end to end vision, so we can expect some bad surprises after the production delivery.

Quadrant 3 Especially functional tests
Here, the projects are properly tested but certainly mainly by functional tests or end-to-end tests. And it is likely that there are mainly manual tests. Applications quality should be fine in production.

However there are 3 disadvantages :
o These are functional tests executed at the end of the development cycle. As there is little testing during development, problems arise at the end of the cycle and tend schedules. This increases the risk that a future version goes into production being not totally stable, e.g. if we do not have time to run all these tests.
o The cost of quality for these applications is important because correcting a bug during functional testing is 10 to 100 times more expansive than correcting it upstream.
o Furthermore, in the future it will be difficult to automate these tests, because they will probably be change sensitive and therefore generate high maintenance costs.

Full article...

Other Resource

... to read more articles, visit

Testing maturity assessment