Testing

[cheese_phantom]

Just curious as always :)

Can it be that testers are not "critical" enough in order to avoid losing their job? That they sometimes just "butter up" the developer, so they are called for testing the next time? Or that thay stay silent in some issues so that they avoid confrontation.

And: do developers like to be confrontated with the awful truth about their games? Do they really listen to good and "merciless" testers? Or just fire them after a while?

[Savant]

It's a balance. The developer wants to hear about all bugs. However, they don't want to hear about the same bug, worded 12 different ways. And they don't want to hear about bugs that were already fixed yesterday (and the tester hasn't got the latest build). And they don't want to hear about a random crash you got one time, that you can't reproduce, and don't remember how it happened.

It becomes a compromise as well. As you get closer to your ship date, you don't want to hear about C level bugs anymore. Just As and Bs. Eventually that becomes just As. And then, "only tell me about show stopper bugs that stop you from playing the game".

To be truly useful, a tester has to have the experience to recognize these stages of development and be flexible enough to work within them.

[Rainer Deyke]

I pay my testers per bug found in order to get the most thorough and harsh testing possible. Even so, I find that testers tends to miss too many bugs. I also have a policy that the game is only released when it is impossible to find further bugs in it, no matter how hard one looks. I am therefore most concerned about C level bugs in the last phases of testing, when the A and B level bugs have all been found and fixed. Despite this procedure, far too many bugs still make it into the released game.

[Phil Steinmeyer]

If I can maybe detour this topic from a potential rant against testers into something more concrete:

I've got volunteer beta testers for my game, and they're very good for certain things (overall feel of the game, suggestions, etc.), but since they're doing this for fun, they're not likely to be as thorough in their testing, or as responsive to my requests, as paid testers.

I don't want to hire a full time tester (at least not directly - too much paperwork hassle), but would probably like to get 2-4 man-weeks of paid testing on my game when it's nearly done. From a couple years ago, I remember that Absolute Quality and the like (contract testing houses), charge ~$35/hour for this kind of thing, which seems perhaps a bit steep to me, as they're likely paying their testers $8-10/hour.

What alternatives have you guys used for short-term, professional testing needs?

[HairyTroll]

I find that testers tends to miss too many bugs.

Do you give your testers a written test plan? Do the testers then create written test procedures for you to review and approve? How do you assure that the testing phase provides sufficient coverage, e.g. do you know if your testers are playing stages that they like and ignoring stages that they dislike?

[Hiro_Antagonist]

I know it's not exactly what you asked, but in my experience, *solid* testers will tend to cost you more like $15-$25 an hour for a year-long contract, in Seattle at Microsoft. (I was paid $20/hour at entry level 8 years ago.) The more short-term the contract is, the more you should expect to have to pay them. If you go through an agency (which handles everything for you and gives you a tester upon your demand), you will pay a premium for that convenience and reliability.

Honestly, $35/hour for a tester that understands what they're doing, has expererience, and is available for an arbitrary amount of time on your demand seems like a very reasonable rate to me. Of course, in your part of the country maybe things run a bit cheaper.

On the other hand, if you're talking about low-end usability testers or button-pushing monkey (the kind MS gets to come in and play xbox games for a day at a time), those guys get pennies and are treated like, well, monkeys. But they also don't know anything about test plans, proper bug reports, solid testing methodologies, etc. In my opinion, those sorts of testers are more like a highly concentrated on-demand beta test team than *real* testers.

-Hiro_Antagonist

[HairyTroll]

The really good testers are good software developers that have, for whatever reason, decided that testing is more interesting than writing code. These testers understand code, so they know what levers to pull and what buttons to press to make a sloppy coders life miserable. That's why really good testers are paid the same as really good developers - at least in my industry.

[cheese_phantom]

If I can maybe detour this topic from a potential rant against testers into something...

I neither have been a tester nor someone who needed testers. I hope the formulation of my questions wasn't too harsh. And I am sorry if some people here that still do testing or had been testers feel offended by them.

I was just wondering how it is possible that after so much testing and developing some games still have very big flaws. So this thing in rhetorics and philosophy, the argumentum fiscilliatorum, came in my mind: do developers or testers stop their inquiry at a certain point and do not ask the questions that seem to be the natural extension of the point of inquiry that they had already reached?

This doesn't mean you need to be a cheater or someone with bad intentions. Sometimes it just means "peace" for everyone if you stop your questions at a certain point. And I wondered if this is also the case in testing. There might be points were the tester doesn't want to risk peace; and sometimes it might be the developer who doesn't want it.

If it is one of the reasons that games come in front of the end-user with flaws and bugs, it should be addressed.

[Phil Steinmeyer]

My experience with testing in commercial game development is that both the testers and the developers are reasonably aware of the game's main flaws at the point it GMs. However, the testing/developing must be cut off at some point due to time/money pressure.

I've seen developers get mad at testers for reporting bugs before, but it's rare. Most developers love good testers, who find legitimate bugs, and correctly classify/filter things (i.e. as Savant says - don't enter the same bug multiple times, don't report minor cosmetic bugs (i.e. 'C' bugs) two days before you're going gold, etc.)

[Hiro_Antagonist]

I was just wondering how it is possible that after so much testing and developing some games still have very big flaws.
I think there are a variety of reasons for this.
-On big games, full test passes are huge, and take forver to execute. They can only be run at major milestones, and sometimes things are changed at the last minute that affect those full-pass test cases that won't be run again...
-Sometimes testers miss things. Especially after they become desensitized to problems that have been around for forever or the game becomes so familiar to them that it's transparent to them. They suffer from that just like developers do.
-Imperfect testing schedules/methodologoes -- Maybe the schedule didn't allow for as much testing as it should have. Maybe the test plan had gaps. Maybe an employee got hit by a metaphorical bus and there was no contigency for someone else to pick up slack. All sorts of things happen...
-Conscious decision -- sometimes bugs are known, but the ship date is around the corner and would cost millions of dollars (plus PR hits) to budge. These decisions are usually made with leaders from the big teams (pm/dev/test) standing in triage, with input from publishers. They have to make hard decisions sometimes, and yes, that sometimes means that shipping the product with a known bug is the best of all options. The customer obviously would choose differently, but they don't have the same factors weighing on their shoulders...

But the short, simple answer is: Human error, imperfect processes, and tough decisions, all of which will always be around... =)

-Hiro_Antagonist

[Phil Steinmeyer]

Resurrecting an old thread here:

Depending on who I deal with for distribution on my next game, I may be responsible for the bulk of/all of the testing for the game.

I already have volunteer beta testers testing my game, but that's sort of a different thing.

For those of you responsible for this yourselves, how do you guys handle 'formal' testing? Who do you work with? Are you happy with them?

[Applewood]

I don't know if its internal or not, but I'm currently working out the bugs in an XBLA title called Novadrome for BVG (disney) and their testing department is f'ing fantastic.

Throughout many years I've often wondered what we were paying the testers for as they've been pretty crap. However, this lot at BVG have not only revised my opinion of external testing a little, but I'm actually in awe of just how thorough they are!