Software QA FYI - SQAFYI

Efficiencies of Virtualization in Test and Evaluation

By: Elfriede Dustin,Tim Schauer

Abstract. Using automated testing in a virtual test environment can reduce the time and effort required to complete test execution and data analysis, significantly reduce test suite costs, and at the same time increase the thoroughness of system testing.

Section 1: Introduction
NIST produced a report in 2002 titled, “The Economic Im -
pacts of Inadequate Infrastructure for Software Testing.” 1

report “estimates the economic costs of faulty software in the U.S. to range in the tens of billions of dollars per year and have been estimated to represent approximately just less than 1% of the nation’s gross domestic product.” The report goes on to state that “based on the software developer and user surveys, the na - tional annual costs of an inadequate infrastructure for software testing is estimated to range from $22.2 to $59.5 billion.” Also in 2004 the Chief of Naval Operations (CNO) Guidance included direction to the Commander, Operational Test and Eval - uation Force (COMOPTEVFOR) to lead a collaborative effort among Navy, OSD, and contractors to reduce the costs of Test & Evaluation (T&E) by 20%. In developing a response to the CNO Guidance for 2004, COMOPTEVFOR surveyed programs and included the following as T&E cost drivers: •

Redundant testing •
Significantly increased levels of regression testing driven by technology insertion •
Increasing complexity of computer software testing, to include systems of systems •
Interoperability testing and certification
Based on the COMOPTEVFOR findings, more effective ap - proaches for testing are needed to be able to meet the CNO Guidance to reduce T&E by 20%.

A GAO Report to the Congressional Committees dated June 2012, describes that “recent defense acquisitions have experi - enced from 30% to 100% growth in software code over time.”

With the increased size and complexity of systems of systems testing, requirements for unique / duplicate test facilities and test-beds for major Navy product areas, software testing is rapidly becoming the “very longest and most expensive pole in the tent” when it comes to fielding new capabilities. Because of many reasons including organizational boundaries, lagging technologies, unique requirements, and testing methodologies, the testing of new capabilities being fielded has become a sig - nificant cost and time element of the process and without some form of change to the current process, could become even mor significant. Some estimates have it consuming more than 60% of the time and cost of the process.

Our experience at IDT shows that using virtual test envi - ronments with automated testing using Automated Test and Re-Test (ATRT) can help reduce testing infrastructure cost for testing areas such as interoperability, system testing, functional testing, component and unit testing. Additional benefits of automated testing in a virtualized environment include a more reliable system, improved testing quality, and reduced test effort and schedule. A more reliable system results from improved performance testing, improved load/stress testing, and improved system development life cycle through automated testing. The quality of the test effort is improved through better regression testing, build verification testing, multi-platform compatibility tests, and easier ability to reproduce software problems. Test procedure development, test execution, test result analysis, documentation and status of problems are also activities ben - efiting from automated testing.

ATRT in a virtual test environment can provide a stable, scal - able, affordable and accessible automated testing infrastructure that extends across one or many server farms, across one or many System(s) Under Test (SUTs) and works with a common set of cloud computing concepts to support a broad virtualized enterprise automated test environment. This specific testing setup allows the use of virtualization in a specialized way to reduce the need for purchasing, storing and maintaining various expensive test environment hardware and software. Proper vir - tualization setup provides a multi-user access automated testing solution that allows users to implement and reuse ATRT, along with all testing artifacts, on a provisioning basis. Additionally all related automated testing activities and processes, i.e. test case and requirements import; requirements traceability, automated test creation and execution, and defect tracking take place in this virtualized environment.

Combining ATRT test efficiency with the hardware cost savings implementing in a virtualized/cloud environment, the resulting estimated savings are tremendous. For example 20 Virtual Machines (VMs) fit on 1 server in our virtual environment example – allowing for huge savings in the test environment, i.e. in this case a 20 to 1 cost savings. Additionally, in the virtual environment, the SUT VM can be located anywhere on a con - nected network and does not need to be located physically in the same VM as the testing VM.

Examples of automated software testing in a virtualized test environment include:
Automatic provisioning of a virtualized automated test environment
Automatic provisioning of the entire automated testing lifecycle for any type of SUTs
Continuous integration using virtualized environments Sections 2.0 through 4.0 provide technical overviews of the various embodiments of the present ATRT/Virtual Test Environ - ment (VTE) implementation.

Section 2: Automatic Provisioning of a Virtualized Automated Test Environment As shown in Figure 1, the virtualized setup allows for a stable, scalable automated testing infrastructure that extends to one or many SUTs or one to many automated testing tool installa - tions (in this example ATRT). This virtualized test environment setup is a highly scalable solution whether a user needs to run 10 or 10000s of tests connecting to N number of SUT displays and servers over days or weeks and whether the user needs to analyze 100s of test outcomes or 10000s or more.

In order to support a virtualized automated test environment, it is critical the automated testing solution itself be scalable. For example, the ATRT technology allows for N number of concur - rent tests to run or N number of serial tests, depending on the test type required. All of the tests and test outcomes are stored in the ATRT database/repository for access by any subscriber (or user) of the ATRT virtualized environment. A subscriber/user can be a developer or tester or anyone on the program with ATRT user access privileges.

This virtualized test environment example setup supports live migration of machines; load balancing; easy movement of machines to different servers without network interruption and allows any upgraded VM to run on any server. As a result, it is also important that in a virtualized environment an automated testing solution is not only scalable but portable. ATRT can test systems independent of OS or platform so it is able to support applications running on both Windows and Linux providing flex - ibility to migrate machines without the constraint of the OS the automated test solution can support.

Additionally, an automated testing tool should be selected that does not need to be installed on the SUT. ATRT is an example of a solution that does not need to be installed on the SUT and instead is communicating with the SUT via a VNC Server or the RDP protocol which transmits the SUT images back to the tester to the ATRT client. Few tools exist that do not need to be installed on the SUT. The typical automated testing tool needs to be installed on the SUT so it can link to the GUI coding libraries to get the object properties of the GUI widgets and/or pull information out of the Operating System’s window manager in order to create an automated test baseline. Installing an automated testing tool on the SUT however is generally not desired, because 1) the installation modifies the system environ - ment (the testing system environment should be identical to the production system environment) and 2) it does not lend itself to cloud computing because of the additional tool installation on each SUT.

In this VTE the SUT VM can be located anywhere on a con - nected network and does not need to be located physically in the same VM as the ATRT VM. This allows for tremendous flex - ibility, for example multiple ATRT VMs can run in the VTE con - necting to 100s of SUT VMs. However, in the typical automated testing setup where the tool needs to be installed on the same machine as the SUT, a 1 : 1 setup is required, i.e. 1 Automated Testing tool for each 1 SUT, negating some of the savings expected in a VTE

Full article...

Other Resource

... to read more articles, visit

Efficiencies of Virtualization in Test and Evaluation