Automated testing

[dntoll]

Background
We are in the middle of the process of betatesting Time Breaker. Most "buggs" are rather easy to fix and after 34 iterations we have eliminated most errors and most of the ones reported are minor issues... But then there is one or two more serious buggs and they are much much harder to reproduce. So we have recently started to think about building a system for automated testing into our games.

The issues
When you introduce new features or refactor your code you normally just test the new features and do not test all features in the game. This means you can introduce new buggs in the game and no one will know before iteration testing (if you even test this then).
During testing you sometimes run in to buggs that are not that common, hopefully you can reproduce the error but if the tester(or her hardware) is not available, you might not be able to reproduce the error.
If you cannot reproduce the error you can only solve it by reviewing the code and guess what happened.

Todays solution
Some issues are pretty easy to discover through normal logging, we log a lot of things but you normally do not log inside the gameloop (for performance reasons) so when some rare thing happens during gameplay you dont have a clue how that happened or how to reproduce it...

the idea
The idea is to record and log every game and be able to reproduce the same game using the recorded data. (randseeds, frametime and inputs). That way we hope to solve these issues. By replaying the game using the recorded data we can hopefully reproduce buggs... It is also easier for the testers who does not have to describe the buggs...
The other thing is that when you ran into some rare bugg and fix it, you can create a new recording of the game where you reproduce the events leading to the bugg and save that loggfile as a testcase. When you introduce new features to the code you just play all testcases to check if the game still behaves normally...

The questions
Do you have any experiences off such a system or have such a system in your games?
How do you collect testing data from your users today? do you have any automated system for that? (ex automated uploading of logfiles, profiling etc.)

(sorry for the language errors, I´m too tired to correct them, phew... :o )

[MadSage]

I've used a number of testing methods over the years. The first isn't really practical for indie developers, and that's having a team of testers constantly testing new versions and adding bugs to a database. The programmer(s) can then mark the bugs in the database as they are fixed.

The most common method I use is logging, even in the game loop! You don't normally have to log every single thing - just key events, such as the player picked something up - then log the object, position, or whatever is required. There is rarely any reason to log every single event in detail. However, the main project I'm working at the moment is a MMOG and we've had to log very detailed information in order to solve numerous synchronization bugs. I already log all packets and network events, but to fix those bugs we needed to log the positions and other information on every frame to pinpoint bugs. To do this, we allocated a huge 40Mb buffer and stored all the required information in binary format. At the end of the game, we wrote it all out to a log file. We were then able to compare the logs from different machines using a compare tool to find where the sync bugs were.

My own game projects will all include a replay system as you talked about, and I've thought about using that method to debug too, if I really needed it.

Another method I'm using involves AI. For example, if you have an AI opponent and your code is flexible, you may be able to get two playing against each other - it depends what kind of game you're working on. Or perhaps you can build an AI player just for testing. This way it's possible to leave them at it for hours, overnight perhaps, to see how solid your code is.

I would always recommend doing some kind of beta test before a release too. Make sure testing is done on many different systems - graphics cards in particular.

Of course, nothing beats careful coding :cool:

[Mark Sheeky]

I used a system as you describe for bug testing Flatspace and it proved invaluable. A simple A.I. "human" player and making sure that each game was exactly reproducable was all that was needed. While running, I accelerated time to a huge degree, allowing over 24-game hours running in a few minutes.

Some errors were incredibly rare and this method found some really unusual bugs that would have (probably) been impossible to find any other way. The hard part is making sure that the autoplay A.I. does enough to locate bugs, and of course you'll miss errors that don't causes crashes (unless you identify what is going wrong and insert a test for them).

Mark

[oNyx]

Developing Your Own Replay System
http://www.gamasutra.com/features/20040204/wagner_01.shtml

Automated Tests and Continuous Integration in Game Projects
http://www.gamasutra.com/features/20050329/roken_01.shtml

Those articles might be of interest.

Oh and (beta) testing is about the most ineffective way to find bugs. What works very well are code audits/reviews (proof reading basically). You can get even rid of glitches which didnt happen yet and the quality of the code gets improved quite a bit, too. Due to psychological side effects you'll start to write cleaner code to begin with. ;)

So, if you have the luxury to do that, do it. Its really worth it. Like unit tests it will save a lot of time in the long run, but it also does work well for smaller projects.

[MrPhil]

Something else to consider: Test Driven Development (http://en.wikipedia.org/wiki/Test_driven_development)

c++ tool for unit testing: cppUnit (http://cppunit.sourceforge.net/cppunit-wiki)

[dntoll]

A lot of good ideas... I think unit testing and code reviews are great! the greatest problem we have had is to bash into our heads that we have to do it!
I think we might have to change our development process in order to make that happen... :rolleyes:

So how many of you are doing code reviews, pairprogramming and unit testing? :D

[ManuTOO]

I used the demo mode of my puzzle-game (recording of all inputs + consistency of the random() function) to remove a lot of bugs.
The recording was always on, so when I was getting a bug, I could try to remove it in the code, and then launch in demo mode and check if it was ok or not... Very very efficient way to do, especially when coupled with the possibility to accelerate the game, to not wait too long time...

[MrPhil]

I am I big believer in Test Driven Development (TDD.) I write code exclusively in this manner both at my day job and my own projects. It has become such an important part of how I write code that it keeps me from using Torque because TorqueScript does not have a test framework tool and I’ve even start writing one!

Using TDD and Unit Tests have many amazing benefits. It drives your code into a flexible and robust design with little up front design effort. Code tends to be decoupled and more compartmentalized. You are less likely to write code and feature you don’t use. You have a massive amount of confidence that changes and additions haven’t broken anything. Refactoring and redesign is effort less and rarely breaks existing functionality. Even hunting bugs has become easier because you quickly hone in on the area where the problem is.

I highly recommend all programmers adopt this methodology.