Software QA FYI - SQAFYI

http:Testing Server Applications

By: Daniel J. Mosley

There are several kinds of situations which the scripts may be designed to invoke during server tests, Volume Testing, Stress Testing, Performance Testing, and Recovery Testing.

The purpose of Volume Testing is to find weaknesses in the system with respect to its handling of large amounts of data during short time periods. For example, this kind of testing ensures that the system will process data across physical and logical boundaries such as across servers and across disk partitions on one server

The purpose of Stress Testing is to show that the system has the capacity to handle large numbers of processing transactions during peak periods. An example of a peak period is when everyone is logging back onto an on-line system after it has been down. In a batch environment a similar situation would exist when numerous jobs are fired up after down time.

As an example, a script might require a user to login and proceed with his/her daily activities while at the same time require that a series of workstations emulating a large number of other system are running recorded scripts which add, change, or delete from the database. The scripts should include activities purposely designed to compete for server resources.

Performance Testing can be accomplished in parallel with Volume and Stress testing because you want to assess performance under all load conditions. System performance is generally assessed in terms of response times and throughput rates under differing processing and configuration conditions. If the tester can identify any business processing cycles (e.g. month-end, Quarter-end, Semi-annual, and annual) the system performance should be tested under emulations of each processing cycle.

Performance testing should cover performance under all hardware and software system configurations. In one particular C-S system it was necessary to test performance under corporate and field environments (Laptops v. desktops, LAN v. WAN), and to test the system in conjunction with a second system utilizing the same server and at times accessing the same database. These are circumstances which can severely impact performance in networked C-S systems.

Hamilton [4] says of C-S performance problems, "And here's another myth that bites the dust: Client/server performance problems can usually be fixed by simply plugging in a more powerful processor." He says this because the performance degradation may be related to other system components not just the processor, and when it is not the processor, adding a box with more horsepower does not solve the problem. He adds, "Your problem may be the network, the computer, the application logic itself. Whatever it is, you'll want to know it before it happens."

He also notes that in his experience the problem is not usually the LAN except in rarest cases. It is most often the result of the client or server being configured inappropriately. Look at cache settings, disk I/O, and network cards first, then at the interaction between the application and the network.

Hamilton believes that the key to successful C-S performance is modeling. He says that modeling allows developers (and testers) to view simulated system components such as Client and server applications, user queues, network flows, etc. before the system is in production.

Hamilton sees the primary problem as a "lack of any kind of established methodology for designing Client-Server performance controls." In view of this deficiency, he suggest attacking performance problems with an "interdisciplinary" team composed of experts in data center operations, network experts, and the IS manager. I would also add a testing specialist to this group.

Hamilton suggests a three step approach. First, have the team decide what is important in your configuration. Second, collect data for those parts of the system. Third, distill the data into usable information. He also says it is important to know the physical performance characteristics (for instance, NT servers have monitoring software which can be used to observe physical performance), and the traffic loads.

The kinds of questions to ask are how much application logic should be remotely executed, how much updating should be done to the server database over the network from the client workstation, and how much data should be sent each in each transaction. Look at all of the processes running on the machine and all of the resources each process receives.

Rossheim [6] suggests a "client-centric approach to Client-Server load management. He advocates putting as much of the processing on client machines as possible (90% or more). Of course, the workstations have to be powerful 486/Pentium processors with plenty of memory (at least 16 mg of RAM).

The best strategy for improving Client-Server performance is a three step process. First, execute controlled performance tests which collect the data points describe in the paragraphs above. Second, analyze the data (MS Excel works very well for charting performance data). Third, examine and tune the database queries, and localize as much data as possible on the client.

If after following the steps above, the performance is not satisfactory, two channels remain open. First, conduct network testing and optimization. Second, purchase more server horsepower. The latter is to be a last resort after exhausting all of the other possibilities.

Regardless of performance testing strategy, it is important to have the right tools when performance testing [2,3] . Mercury's LoadRunner PC, SQA's LoadTest PC can create multi-user tests and SQLEYE, SQL Inspector, and ODBC Inspector can be used during those tests to analysis database performance . These are examples of the kinds of tools which can expedite performance testing. They are discussed in the section on automated tools further down in this article. A big question, however, is what can you do about performance if you don't have any automated tools.

I had the pleasure to work in such an environment. We were charged with testing the performance of a two client-server systems which shared the same server and also shared some database tables (the configuration was an SQL Server database running on an NT server with two processors). Performance was a non-issue until the first system was up and using the server. In fact, I was told to remove the performance test section from my test plan by one of the project managers because we would not have the time or resourses to do it, however, when the response times were miserably slow, performance testing suddenly became a mandate.

The problem was we had no tools and we had to test the performance right away so there was no time to acquire any tools. The approach we took was to write two visual basic driver programs, one for each of the systems. Each driver program randomly initiated each of ten SQL queries designed for it system. The queries ranged from simple to extremely complex.

We began the test with the driver program running on one workstation. Thirty minutes later we executed a second program from another workstation, and then a third and a fourth were started. When we had four users running on the first system we started the first user running on the second system. Thirty minutes later a second simulated user was initiated and so on until we had eight users running simultaneously, four from each system. We then began to logout the users from the first system leaving the second system workstations operative. We did this until there were only the four users from the second system hitting the server. We began logging each one out at short intervals.

The driver programs also wrote the name of each query they ran to an MS Access database table along with the query compile time and the query move time. We imported the data into MS Excel and charted the data. This was sufficient to make determinations about which queries were causing the slow performance. Later examination of the queries revealed that the queries which were degrading performance were poorly constructed as they were requesting a lot more data than was acutually used.

In addition, we determined that a major performance problem was caused by table locking during updates under SQL server. The solution was to update the table in 'batches." The queries were changed so that a small number of rows (250) was updated with each pass, and the query was set to loop until all of the updates were completed. This approach degrades performance a bit, but not anywhere close to the degradation created by locking the entire table. This is a good example of how the limitations of the database technology can affect performance. Future plans are to port the application to Oracle.

There is one flaw in the approach we used. In Client-Server applications many queries are created dynamically. Thus, using a driver program does not always adequately simulate the user interactions via the GUI which would result in the creation of the dynamic queries. Of course, if you manage to identify the circumstances under which queries are created on the fly, you could use a driver program to emulate those situations. The best approach, however, is to use a test tools which employ scripts that interact with the application via the GUI (SQA TeamTest; Mercury Interactive WinRunner) and to execute the scripts on multiple workstations ( SQA Load Test PC; Mercury Interactive LoadTest PC).

Data Recovery Testing is an extremely important type of server testing. Scripts which investigate data recovery and system restart capabilities can save a lot of money and time which could be lost when a production system fails. Recovery testing is even more important because data stored on networked servers can be configured in many different ways. There are several different levels of Redundant Array of Inexpensive Disks (RAID) technology [10] which is a framework for spreading data across several database or file servers. RAID is an approach which is aimed at assuring data recovery if one of the servers fails (RAID is often equated with fault tolerance, but Ruber [7] says this is an over statement of the technology).

Schroeder [9] identified several situations in which data loss can occur. He says data loss can happen when hard disk subsystem failures occur, when system software fails, through accidental or malicious deletion, Destructive viruses, natural disasters, and theft. Sharp [10] sees disks as one of the most likely single points of failure on a system because they are partially mechanical. So, no matter which level of RAID is implemented, the recovery capabilities must be tested because they will be used sooner or later. He says that the recovery time associated with the RAID level you implement should be carefully considered. He also says that there will be a performance degradation during the recovery period. So, you must consider what recovery options a RAID system supports. From a testing perspective, it is important to test all of the recovery options for your RAID system while monitoring system performance.

With respect to system software, Meyer [5] has determined that the majority of Novell Inc. NetWare server crashes due to ABENDs originated by Netware. He also determined that a secondary cause is encountering code which forces NetWare to enter its internal debugger (Break Points). He further identified the three most common ABENDs as GPPE errors, IOPE errors, and NMI errors.

GPPE errors occur when an invalid processor operation is attempted. They are usually the result of a routine which is attempting to access memory that is out of range, e.g. a memory location which is larger than the installed memory.

IOPE errors occur when the server encounters an invalid execution path, e.g. when an execution path points to a memory location which contains data rather than code.

NMI errors are hardware generated errors usually resulting from power fluctuations or RAM failure.

Break Points occur when the stack is corrupted resulting in an invalid return point, or when a function pointer reference an invalid code location.

Thus, server recovery testing should include testing recovery from the four types of errors described above because they are the most likely to occur during the life of a Client-Server system. In some instances, it will be tough to simulate these errors, but a little creative thinking can really go a long way when devising ways to test software. Jerry Durrant (through personal communication) suggests that a simple method to test server crashes is to replace good hardware with hardware which is known to be bad. This practice could be implemented by replacing good RAM with corrupted RAM to test GPPE, IOPE, and in particular NMI errors.

Error trapping capabilities which allow an orderly shut down of the application and not just allow operating system errors to be generated should complement data recovery and system restart procedures Error trapping logic can record the problem, by-pass the corrupt data, and continue processing as an alternative to an system shut down. It is the tester's duty to assure that the system can propoerly trap errors.

Data Backup and Restore Testing is another area of concern. All data base and file servers should have defined backup procedures and defined restore procedures. These procedures should be tested as part of server testing. Schroeder [9] feels it is necessary to simulate many failures from different possible causes in order to test your backup and recovery procedures. He says this is necessary because you want to uncover as many of the vulnerable areas which could threaten data recovery as possible. He says that even with RAID technology backups are still necessary because theft or physical destruction could transpire. Backup and recovery testing can also provide a benchmark for time and resources required when restoring databases.

A backup strategy plan is key to avoiding data loss. Schroeder outlines the issues which the plan must address.

1. How often should the backups be done?

2. What is the backup medium (cartridge, disk)/

3. When will the backups be done?

4. Will the backups be manual or automated?

5. How will it be verified that the backups occur without errors

6. How long will backups be saved?

7. Where will the Backups be saved?

8. How long will it take to restore the last backup?

9. Who is responsible for assuring that backups are done?

10. Who is responsible to do backups and restores if the primary person is not available

These issues along with other issues which may be specific to your environment must be addressed by the backup and recovery plan. Backup and recovery testing should address as many of the issues as possible attempting to find weaknesses in the strategy.

If warm backups are used, the procedures for dumping the database to the backup server, loading the transaction log to the backup server, and for bring the backup server on-line if the primary server goes down for an extended period should be thoroughly tested.

Data Security Testing is extremely significant in client-server systems. Misuse of the database is more of a concern than unauthorized access [8]. In C-S systems data entry and updating is accomplished in a controlled manner through the application interface. Application users have ADD, CHANGE, and DELETE privileges depending their assigned user group. The problems arise from third party software which allows users to login into the database with ACD privileges outside of the controlled C-S application environment. Some mechanism to control access to the database by third party tools is necessary.

Roti [8] suggests several solutions to the third party tool problem. The first is controlled access to third party tools. This is a far from fool-proof solution because even experienced responsible users with commonly distributed database products such as MS Access can attach to tables in production databases and accidentally update them outside of the application. They may not even realize that they changed the database contents.

The second approach Roti suggests is to hide the database driver. For a C-S application, set up a menu choice which loads the database driver, runs the application, and when the user logs out of the application, it unloads the database driver. Of course, sophisticated users can probably find and load the driver themselves. So, this solution is not perfect either.

A third approach, he suggests, is to use stored procedures to control access to specific database tables. All accesses to the database are then routed through a custom data-entry program which calls the stored procedures. When third party tool users try to login and update the database they find that they do not have ACD privileges. The hole in this approach is that some third party tools have the ability to run stored procedures.

The fourth method Roti suggests is to use encrypted passwords. When the user enter his/her password it is encrypted be for being sent to the server. The user is unaware of this and sees only his/her login ID and the unencrypted password. Third party software tools not have the encryption algorithm would not be able to access the database or could access it with read only privileges.

The fifth solution is use shadow user names. All users login with read-only access while their privileges are checked. If they are allowed both read and write access, they are re-connected with a shadow user name which they do not see. This would allow the use of third party software packages, but with read only access.

Regardless of the security mechanism, testing the security of C-S system databases is absolutely essential. Controlled tests with third party tools should be conducted separately or as part of the system test. In addition, tools such as SQLEYE [11] can be used to monitor and restrict database access with third party tools.

Full article...


Other Resource

... to read more articles, visit http://sqa.fyicenter.com/art/

http:Testing Server Applications