|
Efficiencies of Virtualization in Test and Evaluation
By: Elfriede Dustin,Tim Schauer
Abstract.
Using automated testing in a virtual test environment can reduce
the time and effort required to complete test execution and data analysis,
significantly reduce test suite costs, and at the same time increase the
thoroughness of system testing.
Section 1: Introduction
NIST produced a report in 2002 titled, “The Economic Im
-
pacts of Inadequate Infrastructure for Software Testing.”
1
This
report “estimates the economic costs of faulty software in the
U.S. to range in the tens of billions of dollars per year and have
been estimated to represent approximately just less than 1% of
the nation’s gross domestic product.” The report goes on to state
that “based on the software developer and user surveys, the na
-
tional annual costs of an inadequate infrastructure for software
testing is estimated to range from $22.2 to $59.5 billion.”
Also in 2004 the Chief of Naval Operations (CNO) Guidance
included direction to the Commander, Operational Test and Eval
-
uation Force (COMOPTEVFOR) to lead a collaborative effort
among Navy, OSD, and contractors to reduce the costs of Test &
Evaluation (T&E) by 20%. In developing a response to the CNO
Guidance for 2004, COMOPTEVFOR surveyed programs and
included the following as T&E cost drivers:
•
Redundant testing
•
Significantly increased levels of regression testing
driven by technology insertion
•
Increasing complexity of computer software testing, to
include systems of systems
•
Interoperability testing and certification
Based on the COMOPTEVFOR findings, more effective ap
-
proaches for testing are needed to be able to meet the CNO
Guidance to reduce T&E by 20%.
A GAO Report to the Congressional Committees dated June
2012, describes that “recent defense acquisitions have experi
-
enced from 30% to 100% growth in software code over time.”
2
With the increased size and complexity of systems of systems
testing, requirements for unique / duplicate test facilities and
test-beds for major Navy product areas, software testing is
rapidly becoming the “very longest and most expensive pole in
the tent” when it comes to fielding new capabilities. Because
of many reasons including organizational boundaries, lagging
technologies, unique requirements, and testing methodologies,
the testing of new capabilities being fielded has become a sig
-
nificant cost and time element of the process and without some
form of change to the current process, could become even mor
significant. Some estimates have it consuming more than 60%
of the time and cost of the process.
3
Our experience at IDT shows that using virtual test envi
-
ronments with automated testing using Automated Test and
Re-Test (ATRT) can help reduce testing infrastructure cost for
testing areas such as interoperability, system testing, functional
testing, component and unit testing. Additional benefits of
automated testing in a virtualized environment include a more
reliable system, improved testing quality, and reduced test effort
and schedule. A more reliable system results from improved
performance testing, improved load/stress testing, and improved
system development life cycle through automated testing. The
quality of the test effort is improved through better regression
testing, build verification testing, multi-platform compatibility
tests, and easier ability to reproduce software problems. Test
procedure development, test execution, test result analysis,
documentation and status of problems are also activities ben
-
efiting from automated testing.
ATRT in a virtual test environment can provide a stable, scal
-
able, affordable and accessible automated testing infrastructure
that extends across one or many server farms, across one or
many System(s) Under Test (SUTs) and works with a common
set of cloud computing concepts to support a broad virtualized
enterprise automated test environment. This specific testing
setup allows the use of virtualization in a specialized way to
reduce the need for purchasing, storing and maintaining various
expensive test environment hardware and software. Proper vir
-
tualization setup provides a multi-user access automated testing
solution that allows users to implement and reuse ATRT, along
with all testing artifacts, on a provisioning basis. Additionally all
related automated testing activities and processes, i.e. test case
and requirements import; requirements traceability, automated
test creation and execution, and defect tracking take place in
this virtualized environment.
Combining ATRT test efficiency with the hardware cost
savings implementing in a virtualized/cloud environment, the
resulting estimated savings are tremendous. For example 20
Virtual Machines (VMs) fit on 1 server in our virtual environment
example – allowing for huge savings in the test environment,
i.e. in this case a 20 to 1 cost savings. Additionally, in the virtual
environment, the SUT VM can be located anywhere on a con
-
nected network and does not need to be located physically in
the same VM as the testing VM.
Examples of automated software testing in a virtualized test
environment include:
1.
Automatic provisioning of a virtualized automated test
environment
2.
Automatic provisioning of the entire automated testing
lifecycle for any type of SUTs
3.
Continuous integration using virtualized environments
Sections 2.0 through 4.0 provide technical overviews of the
various embodiments of the present ATRT/Virtual Test Environ
-
ment (VTE) implementation.
Section 2: Automatic Provisioning of a Virtualized
Automated Test Environment
As shown in Figure 1, the virtualized setup allows for a stable,
scalable automated testing infrastructure that extends to one
or many SUTs or one to many automated testing tool installa
-
tions (in this example ATRT). This virtualized test environment
setup is a highly scalable solution whether a user needs to run
10 or 10000s of tests connecting to N number of SUT displays
and servers over days or weeks and whether the user needs to
analyze 100s of test outcomes or 10000s or more.
In order to support a virtualized automated test environment,
it is critical the automated testing solution itself be scalable. For
example, the ATRT technology allows for N number of concur
-
rent tests to run or N number of serial tests, depending on the
test type required. All of the tests and test outcomes are stored
in the ATRT database/repository for access by any subscriber
(or user) of the ATRT virtualized environment. A subscriber/user
can be a developer or tester or anyone on the program with
ATRT user access privileges.
This virtualized test environment example setup supports
live migration of machines; load balancing; easy movement of
machines to different servers without network interruption and
allows any upgraded VM to run on any server. As a result, it is
also important that in a virtualized environment an automated
testing solution is not only scalable but portable. ATRT can test
systems independent of OS or platform so it is able to support
applications running on both Windows and Linux providing flex
-
ibility to migrate machines without the constraint of the OS the
automated test solution can support.
Additionally, an automated testing tool should be selected
that does not need to be installed on the SUT. ATRT is an
example of a solution that does not need to be installed on the
SUT and instead is communicating with the SUT via a VNC
Server or the RDP protocol which transmits the SUT images
back to the tester to the ATRT client. Few tools exist that do not
need to be installed on the SUT. The typical automated testing
tool needs to be installed on the SUT so it can link to the GUI
coding libraries to get the object properties of the GUI widgets
and/or pull information out of the Operating System’s window
manager in order to create an automated test baseline. Installing
an automated testing tool on the SUT however is generally not
desired, because 1) the installation modifies the system environ
-
ment (the testing system environment should be identical to the
production system environment) and 2) it does not lend itself to
cloud computing because of the additional tool installation on
each SUT.
In this VTE the SUT VM can be located anywhere on a con
-
nected network and does not need to be located physically in
the same VM as the ATRT VM. This allows for tremendous flex
-
ibility, for example multiple ATRT VMs can run in the VTE con
-
necting to 100s of SUT VMs. However, in the typical automated
testing setup where the tool needs to be installed on the same
machine as the SUT, a 1 : 1 setup is required, i.e. 1 Automated
Testing tool for each 1 SUT, negating some of the savings
expected in a VTE
Full article...
Other Resource
... to read more articles, visit http://sqa.fyicenter.com/art/
|