Software QA FYI - SQAFYI

What to Test When It's Not Your Code

By: Ipsita Chatterjee

In today's world, where outsourcing software development is the trend of most companies, it is important for customer-oriented test analysts to analyze the vendor's application and carefully decide what needs to be tested. In most cases, the application bought from the vendor needs to integrate with the legacy systems of the organization in some way. Only in a handful of cases can the purchased application be used as a standalone package to meet certain business requirements.

So what should an organization test when buying an application from a vendor, and what factors need to be considered when making this decision? I have consolidated a few guidelines to answer these questions. The list is not complete; you should add new experiences from people in your organization.

The Purpose of the Purchased Application Projects that involve buying an application and making it work with existing legacy systems usually do not have clear and concise requirements, even if a set of business requirements were used when making the decision to buy the application. There are other mandatory requirements that need to be considered in order for the application to meet user expectations.

For example, if the new application requires data from a warehouse of legacy data, what format must the data be in for it to be accepted? Can it be loaded as a CSV file, or does it need to be in an XML format? Is the range of data available to cover the business requirements, or does the source data have to be manipulated and manually fed into the third-party application?

Two other important things to consider are the organization's functions or business processes that the application will manage and how these will be fulfilled when the legacy systems are integrated with the third-party application. These form the basis of the requirements that should be used for testing.

In other words, it is a good practice to test the requirements of the solution from an end-to-end perspective rather than only from the business requirements standpoint.

Is the Outsourced Application a Black Box Component? Has the vendor thoroughly tested its application? Can we treat it as a black box? Does it fit as a component within our system and work in an end-to-end fashion? Being a test analyst I would like to answer yes to all the above questions, but often the answer is no. Even if the purchase contract states that the application should be tested by the vendor and should have no critical defects at the time of integration, such documentation cannot guarantee that this will be the case. History shows us that applications are delivered with considerable amounts of defects. This may not be due to lack of testing, but rather to missed or misinterpreted requirements.

If the application is envisaged for use in complex calculations, it adds to the risk. What if the application is calculating things incorrectly? If the assumptions made in the mathematical modeling are wrong or out of line by even a fraction, the impact on the business may be severe. Ultimately the organization is liable for these mistakes. The severity of the impact of existing defects in the outsourced software on an organization should determine whether or not the software can be treated as a black box.

In most cases, we cannot afford to treat it as a black box. So what is the solution? I have dealt best with this kind of a situation by testing the vendor's code as a standalone application before integrating it with our legacy systems. This has served a two-fold purpose: I ensure that the outsourced code is of acceptable quality before integration; and issues that arise during integration are easily separated into integration issues from the legacy system and defects in the vendor application.

It is also a good practice to test the output of the legacy system that will be fed into the vendor application. Breaking down the testing phases saves a lot of time and rework in later stages.

Comprehensive User Acceptance Testing at the Correct Time In most places I have worked, the purchased application was subjected to thorough testing by the end-users. But this may not have been the correct time to test. So, when is the best time to bring the users into the picture?

Most testing models say that an application should be subjected to user acceptance testing after system testing--a concept that may hold for systems built in house.

I have realized that the best people to judge quality of outsourced solutions are those who participated in the proof-of-concept phase with the vendors. That said, the users should participate in testing the vendor's code as a standalone application. This will help address the defects caused by missed and misinterpreted requirements made earlier in the lifecycle and can be addressed before the integration takes place. The end-users should also test the application once the integration and system test are finished, so as to cover the business process testing in an end-to-end fashion and add more value to the testing.

The test approach and the content of what is tested in the vendor's software applications has to be radically different from the traditional testing approaches and models. It is difficult to set specific guidelines for testing third-party applications, especially when user acceptance testing alone is not sufficient to achieve the level of quality we want in the code. As outsourcing becomes more popular, we will have to change traditional testing approaches in order to meet the quality requirements of different organizations.

The next part of this article will cover what not to test when testing someone else's code.

Full article...

Other Resource

... to read more articles, visit

What to Test When It's Not Your Code