Software QA FYI - SQAFYI

State of Mobile Testing 2013

By: lan Trefzger

Are you doing enough mobile testing? Most don’t, find out how you compare to other development and testing teams. XBOSoft surveyed global software development and software testing teams to understand mobile testing best practices. Use the results from their report to benchmark your own efforts. Below is an article based on the part of the report that covers what types of testing organizations do.

Mobile Testing Practices

Testing practices range from, what type of testing is performed, to localization testing in different countries; from how many types of devices you test on to testing on different networks. Many of these examples reveal how much more complex mobile devices are to test than desktop platforms.

What type of mobile testing do you do?
Mobile Testing Practices
Only 43% do any performance testing. This should be 100%. Letting your customers find your performance problems can be very expensive. For example: Google found that for every 1 second increase of page load time, a 20% loss in customers was recorded.

The challenges for performance of mobile testing can be divided into three parts:
1. Mobile App testing on the device [1]
2. Network issues
3. Server performance.[2]


For an Agile environment, many contend that regression testing can’t be done for each cycle unless you use automated testing. For an Agile team that has a production release each month, there is only so much time that can be spent on testing with such a short time frame, so only new features are tested. Regression testing of all those features and functions which shouldn’t have been affected is often neglected. Automation is the only way to solve this dilemma.
What type of platforms do you support?
What type of platforms do you support?
Android and iOS continue to dominate with both having over 80% of the developer’s support, which tracks with most other surveys.

Windows has 41% support, which is very surprising since Windows market share is in the low single digits and 10% are developing for Windows to the exclusion of either iOS or Android. They must believe market research company IDC’s forecast report that Window’s market share will rise to 20% in the next few years and it is hard to believe that that the Nokia-Microsoft alliance won’t be successful. Window’s developers are half as likely to have a dedicated development or QA team.

How many different devices is your SW used on (approximately)? Please count one device that is used with 3 different OS versions as 3 devices.

How many different devices is your SW used on (approximately)? Please count one device that is used with 3 different OS versions as 3 devices.

How many actual devices do you test on? Please count one device that is used with 3 different OS versions as 3 devices.

How many actual devices do you test on?

Most mobile developers test with a small set of devices. Unfortunately, the number of devices on which your application will run is staggering. Even with testing on more than 24 devices (one company tested on 200 mobile devices), you are testing on only a small fraction of the devices that your mobile application will be used on. Since it is impossible to test on the full universe of devices that your application will be used on, you have to test smart and choose a minimum set of devices that gives you maximum coverage. This is not easy since that set of devices will change from quarter to quarter. This is where 3rd party QA experts can really help. They have sets of devices for different types of application and are continually updating them. They can match you up with one of their pre-determined sets of devices, and then modify and add to it based on your specific requirements.

Full article...


Other Resource

... to read more articles, visit http://sqa.fyicenter.com/art/

State of Mobile Testing 2013