Monday, July 16, 2012

Soak it up

Soak testing is a brute-force method for software testing by running the application over a long period of time. It is effective for tracking down memory leaks, rare timing crashes and obscure bugs.

For games, the key is to find a simple method for running most of your game code without any user input. To emulate player behavior we create a simple AI that approximates a player, but simplifies some aspects to streamline the game cycle.

Lets consider how we might do this in a 2D top-down shooting game. We create an AI that moves randomly around the stage and fires it's weapons randomly. The player health is forcibly increased so that he does not die but such that receive-damage processing is still performed. We choose weapons randomly to cover all cases of enemy and weapon collision over a long enough time period. There is not so much complexity in a 2D shooter so this would generally suffice to cover all activities a player might perform. Any special abilities or similar can be randomly triggered as necessary.

For a first person shooting game, more complicated AI is necessary, but we should still simplify where appropriate. Similarly we can choose random weapons and shoot randomly at enemies (with some variance to emulate player inaccuracy). We can simplify by forcing player rotation and position. To get good map coverage we can simply pick random locations of interest (items, enemies, checkpoints) and fly along path-finding routes to get to them.

It is important to simplify past areas that are going to be likely to result in 'stuck' situations. These situations are the biggest hindrance to effective soak-testing, so brute-force anything necessary to get past them.

Providing a way for your game to run at a higher speed allows for much more effective testing. You can disable vsync or frame-time waits and have your game run as fast as possible. Core rendering can be disabled to speed things up much more, but this will of course reduce testing of the rendering system. Reducing screen resolution to a few pixels is another easy win depending on your performance bottlenecks.

A further simplified stage-cycling system can get a lot of coverage with little programming effort. Setup your game to load a random stage, wait a second and then do it again. This is a good first step for your soak-testing system, and can catch a surprising number of bugs. Try hooking this up to run automatically after every build commit to catch some errors in a timely fashion.


  1. On project I can't name (now cancelled) we had something like this. However since it was a multiplayer game our soak also included lobby join/leaves for the game.

  2. Even a simple integration test is better than just unit tests is better than no tests.

    Soaking, as Kalin proposes is the best testing as it exercises more code in the right time frames.

    A warning: soak tests are hard to pass, and hard to debug unless you have extensive logging, which can and will impact on your target image. So, best to log via network using a kernel that has been tested separately and has a small memory footprint.