Gameplay Recording & Playback for Observing Beta Testers

[Martoon]

Most of us know how valuable it is to watch someone play your game (while resisting the urge to prompt or help :) ), especially someone playing for the first time. You can learn a tremendous amount about how players approach the game, what's intuitive and what's confusing, which features they like and use the most (and which features they never discover), etc.

Unfortunately, you can quickly run out of friends to test the game on, especially people who haven't already seen it. It's usually more practical to enlist the help of beta testers over the internet. These remote testers can volunteer written feedback, but there's a lot of info you miss.

It seems it could be very useful to implement journaling and playback, including mouse motion through menus, etc. You could then have your testers send you their recordings, starting from the very first time they launch the game.

If you wanted to go all-out, you could even offer the opportunity to plug in a microphone so they could record any thoughts or comments as they played. This could be compressed to a low bitrate .ogg file and included (and synched) with the gameplay recording.

Just wondering if anyone has implemented this, encouraged their testers to consistently send back their recordings, and what kind of benefits they saw?

[Jim Buck]

Check this (http://www.joelonsoftware.com/articles/UsabilityTestingwithMorae.html) out.

[Donavon Keithley]

I'm a big believer in this sort of thing. It's also great for reproducing bugs, testing code, and in-game demos (like how Professor Fizzwizzle! uses input playback to show you the solution to the level).

It's tricky in that you have to identify *all* inputs into the system such that, given those inputs, the system is then 100% deterministic. Besides controls, time is the other big input that has to be journaled. File I/O is another potential source of variability. Game assets aren't usually much of a concern but you might want to journal data from configuration files, registry keys, and such, and of course network I/O if any. Multiple threads can be anything from a non-issue to a huge problem, depending on how you use them.

For beta testing, maybe journal off the microphone input too (with the user's permission of course :)). That would actually be a great way to get feedback -- very easy for the player and you get to hear their peals of joy and howls of frustration.

And hey, while you're at it, add web cam to that and you've almost got a system like Morae that Jim linked to. :cool:

Unbridled ambitions aside, obviously changes to the system will often invalidate journals so version control is essential.

Another thing to consider is journaling subsystems. I've done this with graphics for instance. It makes a much fatter journal but they're less fragile in the face of changes to the game and play back much faster. Great for development.

(A company called Replay Solutions (http://www.replaysolutions.com) has some technology that automatically instruments your binary to journal off & play back the inputs. Very cool stuff, but priced out of the reach of most indies, I'd imagine.)