Different Testing Approaches for Mobile Games
By: Ville-Veikko Helppi
The thoughts and opinions expressed are those of the writer and not Gamasutra or its parent company.
One of the most common problems in mobile game testing – similar to app testing – is how to access all possible devices that your game users are using. That said, despite thousands of unique models already out there for Android (and with all possible iOS variants with different OS versions) only few hundred of them are really relevant in the big picture.
Let's take a look at different mobile game testing approaches, as well as infrastructural and architectural aspects of the mobile game testing...
Architecture and Infrastructure Aspects of Mobile Game Testing
Let’s start from software architecture. Many of today’s mobile games are based on open source or commercial game engines, such as Unreal, Unity3D, CryEngine, Construct, PlayCanvas, Cocos2D, and many others! (DISCLAIMER: I’m not taking sides of which game engine works the best or delivers the greatest features, just listing few popular ones for reference.) Furthermore, many of these game engines provide great tools and frameworks to get you game quickly built, with all those nice graphical features.
Looking this from the traditional software point of view, this is like a “tool – app – middleware” offering that these products will provide you, as well as compiling your game to specific platform (many of these products are cross-platform and enable you instantly to compile the game for Android, iOS and other platforms as well). For the platform part, let’s use Android as an example: an “open” operating system with a stack of software components which can be roughly divided into four layers: Application, Application framework (manager layer for content, resource, package etc.), Libraries (e.g. OpenGL ES software, Fonts, WebKit, SGL etc.) and the Linux kernel (drivers for graphics, audio and keypad, power management etc.). In addition, platform here includes the actual hardware running whatever chipset on it, with GPU and displays of different resolutions.
Architecture and Infrastructure for Mobile Game Testing
After that your mobile game needs to communicate with back-end servers using your own or Google/Apple/other services via WiFi, radio or some other type of communication channel. The need for testing the connectivity between your mobile game and back-end services is nowadays very important. For example, ads are shown via connectivity and without those running properly on your device, you are taking an extra risk in front of your core audience – mobile gamers. Sure, they would probably like it better without ads, but from the monetization point of view, you surely want to test this thoroughly before submitting your app to app stores – and make sure you are not losing revenue because of faulty implementation.
As said, testing connectivity should be one of the key things to do when building mobile games. Test automation can bring you a lot benefits for automating the connectivity testing, and scaling the infrastructure from one user to hundreds of thousands of users.
Testing Hermetic Environment
Today, nearly all mobile games have a server-client interaction, some requiring a login, uploading of data (e.g. game results/progress) and downloading of data (e.g. ads). When adopting these kinds of services to be part of your games, developers need to ensure that all their changes done in code, to either server side or client side do not break the functionality of the service. Furthermore, this type of environment can be also set up on a local machine/server where critical connectivity and interaction between user and server can be tested. In fact, testing for some of these types of use cases could be done efficiently with a hermetic environment – however, not jeopardizing the integrity of your game.
Hermetic Environment Testing
Hermetic testing can be seen and done as easily as switching your device to flight mode. Nothing gets send out and nothing gets in.
Testing Compatibility - and Doing Porting
When your mobile game is getting developed, you should always test on the real platform that the end-users are using as well. This means that you must test your game on real devices. Emulators won’t simply provide you as authentic environment as it would be required to get full understanding of real behavior in hands of real users. What makes a difference between using an emulator or real hardware? Typically, the lack of real hardware is slowing down and forces developers to use emulators, but this can shrunk into 4 categories why games should be only tested on real devices:
1) SOFTWARE. It’s just not only about vanilla platform software that makes the difference, but nowadays OEMs customize their software a lot: UI layers, customized legacy software, only certain widely used apps are pre-integrated with the device (e.g. Twitter, Facebook).
2) HARDWARE. There are LOTS of differences in chipsets, GPUs, even connectivity hardware, displays, sensors, and memory. This is very hard to emulate appropriately – and even it would get as close to real world scenario – all of that is executed on PC x86 hardware, and not ARM-based hardware.
3) USER EXPERIENCE. This is heavily related to performance. The graphics performance of the system makes the user experience, either good or bad. And there is an absolute must to get mobile game tested on real hardware.
4) NETWORK. The system can experience network issues and slow network is not possible to be tested on emulation. In terms of network configuration, emulators run on the PC, connect to the LAN and access the internet via your corporate firewall. Using real handsets, the network is connected to the radio interface and from there to the internet.
Whatever platform (e.g. native mobile platforms, web etc.) you are building your mobile game on, remember that only real devices give you real results. Furthermore, the depth of testing (e.g. how many devices are used, how many different configuration those cover) is important as you don’t want to minimize your target audience who have potential to play your game – make you money!
Smoke and Reference Testing
When the mobile game is in the state that is testable in the context of 1) game itself, 2) platform-compiled and 3) connects with back-end services (or just platform APIs) the full-blown reference testing can give great results and understanding of how well game does. But even much earlier than that, it is possible to start doing smoke testing and using even hundreds of real Android/iOS devices for that. When bugs are revealed and fixed earlier, it will save you a lot of time (and money) and gets you earlier to publish your game for gamers. In addition to regular smoke and reference testing, it’s a good idea to include real performance and battery drain tests as part of your testing pattern.
The reference testing gives great understanding of how your game does, but it doesn’t cover everything. For example, huge number of users can quickly take down the back-end, slow down the game-play and even worse, force users to stop playing the game. More about this in the next chapter…
Stress, Load, Spike, Soak – Performance Testing
Good performance is very closely related to good user experience. User wants to see constant progress with the game, do the smooth game-playing, graphics performance needs to be up to snuff and work across all different mobile devices (phones, tablets etc.) the very same. In addition, it is important to understand how well back-end(s) can handle massive number of users and significant load.
... to read more articles, visit http://sqa.fyicenter.com/art/