Dont you get scared at release?

[zoombapup]

I was reading about the failure of Elemental at release over at rockpapershotgun

http://www.rockpapershotgun.com/2010...-release-fail/

It made me think of something I hadnt really considered much yet. That is, the release testing!

It scares me silly that I'd release something that literally couldnt be played by anyone but me. Although I like to think I've got a fairly pragmatic design sense at least. I've got some fair resources (in the form of my students) when it comes to basic testing, but as anyone whose released a major game will tell you, complex games have a LOT of bugs. Hell we used to have a pretty big test departnment and still shipped with bugs. Now its just me I reckon I'll have kittens when trying to release my game.

How do you guys handle testing cycles? Do you have alpha/beta tests? Do you have a significant number of testers? Do you find you have lots of bugs on release? If so, how do you handle the fallout of those bugs?

I've always thought that having a "soft launch" would be the way to go, but I've heard plenty of people now saying that isnt such a great idea. So maybe instead it should just go into a long beta with a fairly limited set of testers instead?

Thoughts please?

[Desktop Gaming]

I didn't really do much alpha testing for Magicville - only two people other than me saw it before Beta. At Beta I had I think seven people plus me testing and they found a lot of stuff - the game was played from start to finish twice over by one person, plus the other testers trying to break it, before I was happy to let it go. It was in beta for about six weeks.

When I released Magicville it had a doozy of a bug in it, caused by third party code.

The code I used attempted to set permissions for the "Users" group... the code referenced this group by name, not its constant. This turned out to be very, very bad, as the group isn't called "Users" on any non-English OS, causing the whole game to fail.

How did I miss that? Well, I'm English and have an English OS! I didn't see it coming in a million years but come, it did. I got a revised build out within a day but it still took a few days to filter through to market. My conversion rate took a nasty hit (the bug-free version's conversion rate was 3x that of the first release). I don't like to think about how much money it cost me in lost sales.

How will I stop it happening again? It probably won't, now that I know! But even so, I have the German Edition of WinXP running on my Mac via VMWare.

When people find post-release bugs I try to be apologetic and positive in getting it quickly fixed. People usually understand!

[Applewood]

I think if you do some basic research into hiring a proper QA studio based in India, you'd be absolutely stunned how cheap it is. I mean, seriously!

[cyodine]

A friend of a friend also swears by the effectiveness of India's tech support.

As for QA, I'm wondering if India's cheap QA is more of a test against bugs, or whether they have companies with experienced gamers that can give sound feedback on the gameplay as well. Does anyone know?

And speaking of exploiting / enriching (however you want to view it) the third world, anyone know much about Samasource? I think it's IT services based in Africa.

As for release bombs, I guess it does kind of scare me a bit, although only slightly since there's no social expectation for my first release to be great; if it bombs it'll probably just be viewed as the status quo. I won't have tons of negative publicity; I'll have just none.

I guess frequent tester feedback throughout the development process is essential to avoid "blindness". I'm wondering if many of you privately network with those you trust in this forum to get sound outside feedback on your game's progress without the much greater risk of having your ideas snatched (current separate forum thread) via just exposing all your work to the general public.

[Applewood]

A friend of a friend also swears by the effectiveness of India's tech support. As for QA, I'm wondering if India's cheap QA is more of a test against bugs, or whether they have companies with experienced gamers that can give sound feedback on the gameplay as well. Does anyone know?

My involvement has been to have a game we wrote for someone else passed through an Indian company, and we just had feedback about the usual bugs and stuff.

I was impressed by how thorough it all was and they caught a variety of things that wouldn't show up by just going through the motions. Repro steps were clear and in gud Inglund.

Apparently the bill was 500 quid and that included the complete pass of bugs and then a full regression test a week later when we'd allegedly fixed them all.

Regarding getting general gameplay feedback, some might off that but I'm not interested tbh. It's better to trial your stuff on a small group of the intended audience and you can usually get that done for the price of half a dozen pizzas.

Regarding "abuse" of the 3rd world, that's not really something I have a problem with and don't consider it to be abuse. Technically, India is in the 3rd world as it's not a part of Nato or the Warsaw Pact, but it's a pretty solid country with good people and a democratic government and that's all I need to know.

From my perspective, the only reason not to outsource to foreignors revolves around depleting the jobs market at home but that's something I'll spend more time worrying about once my own company is in the fortune 500 tbh.

[princec]

We have a whole room full of Indians here at my current employed. Suffice to say I don't think they're worth the electricity they use.

Here at Puppy Towers I spend approximately 50% of game development time testing. It's a relatively expensive way to do it but as I'm also the producer I'm fairly sure that I get what I want

Then we tend to release betas quite early to pick up the technical troubles. We also log all exceptions and failures automatically and every point release I sort them in order of incidence rate and fix the top 10. This has done wonders for quality control.

I would like to find a cheaper mechanism for testing than unit testing or using my time. Unit testing is hailed in the business world as some sort of mantra but my empirical observations is that unit testing effectively doubles development time and catches very, very few bugs.

Cas

[Applewood]

Total agreement on unit testing - just the latest emperors new clothes.

You shouldn't do your own gameplay testing though, seriously. You get so used to what you're looking at and checking that what you did actually does, that the bigger picture is easily missed and often even forgotten.

You'd probably never notice there isn't a way to quit the game properly because you left in the <esc> bugs out immediately hack for example.

Regarding your room-full of Indians, there's nothing inherently brilliant about your average Indian citizen. I recommend them purely because they're phenomenally cheap when they're in India!

[ChrisP]

You'd probably never notice there isn't a way to quit the game properly because you left in the <esc> bugs out immediately hack for example.

Funny story: During much of Mayhem Intergalactic's development, there was a crash bug that occurred with 100% reliability whenever you started a game, quit out to the menu, and then started another game. Pretty easy-to-find bug, right?

It went unnoticed for months. Because every single time I tested the game, I would start a new game, play it for a while, and then quit the entire app to make changes and recompile before starting a new game. I was so narrowly focused on testing the "actual game" that the idea of testing other sequences of menu button-presses completely escaped me. When I did blunder into the bug one day, it was completely by accident.

These days I try to be a little more comprehensive, but you can never be sure that you've found everything yourself. So getting as many eyes as possible on the product before release is an absolute must, however you choose to do it.

I can echo DG's comment about non-English versions of Windows, that tripped me up as well. Though in my case it was broken Unicode handling in some file loading code (passing a UTF-8 string into the ASCII version of a Windows function... doh).

Oh, and then there was the time I updated Steam achievement progress every time you performed the game's core action... which triggered popups that built up, spammed the screen and neatly obscured vital parts of the UI for minutes at a time. Meaning the Steam version of the game was essentially unplayable, despite extensive pre-release testing. You see, those popups don't actually appear until the game is publicly available via Valve's servers. Good times.

So yeah, I'd say release is pretty scary.

[princec]

Total agreement on unit testing - just the latest emperors new clothes.

You shouldn't do your own gameplay testing though, seriously. You get so used to what you're looking at and checking that what you did actually does, that the bigger picture is easily missed and often even forgotten.

You'd probably never notice there isn't a way to quit the game properly because you left in the <esc> bugs out immediately hack for example.

Regarding your room-full of Indians, there's nothing inherently brilliant about your average Indian citizen. I recommend them purely because they're phenomenally cheap when they're in India!

Yes this is probably S*ny's mistake - importing them over here and having to pay 10x the amount from them. Most of which goes to the p1mps.

Doing my own gameplay testing is definitely a bit of a bummer but I put on a player hat most of the time when I'm doing it. Unfortunately means I get sucked into a playtest session for an hour. Sometimes I test things out on the girlfriend

Cas

[Dan MacDonald]

I'd love to understand more what went wrong with Elemental. It sounded more like the game mechanics were broken then technical issues. I've always admired Brad Wardell and I do appreciate his honesty in criticizing his own company. I really didn't know what to do with his justification.

EVERY competent software developer knows that the programmer must never be the one deciding whether the program is done. Yet, my love of Elemental broke my self discipline and I began coding on the game itself in vast amounts and lost any sense of objectivity on where the game’s state was. I normally only program the AI on our games so I can keep a level of distance from the game itself to determine whether it’s “Ready”. On Elemental, I was in love with the world and the game and lost my impartiality.

It sounds legitimate enough, at the same time this is the reality of developing indie games for most indie developers. In most cases one individual is doing all the programming and design and able to maintain objectivity and deliver quality, the competent ones anyway. So I have to disagree with his justification. I think there are some cases where his statement is true, but it's not true of most successful indie developers I know.

[zoombapup]

Its definitely a trap that many can fall into though Dan. I mean its always brutal when you put a game you think is finished in front of real players and they just flounder around looking confused.

I've seen grown men almost in tears when faced with a player that literally cannot understand what to do. It seems so obvious when you are developing the thing, but in the cold light of someone else's experience things just arent percieved the same.

I guess I have to hope that my Uni ties will see me through a half decent round of testing at least. I suggest that others do the same (find a local university with a game dev course and approach them to do some testing for you, they will be surprisingly happy I would suspect).

Plus for the most part students are often genuinely wanting to help improve the product too.

[Bram]

When it comes to release, I get really paranoid, and check, check and check again. This saved me from most big screw ups.

The biggest screw ups so far for me were:

(1) I released a game which I had tested on iPad/iPhone but not iPodTouch. The latter crashed with black screen. I only had .xib files for former platforms.

(2) I wrote my own leaderboard code, which managed to poison all clients: When one user used an exceptionally long gamername, it got stored by my leaderboard server, however the server truncated the string halfway through a unicode.
When the server then sends to all users: all users would crash: my game was no longer playable for anyone with internet connection.

The first bug was really mitigated by the fact that it was a free game with in app purchases: this means that not a single paying customer was affected!

So my advice: make game free, use in-app-purchases: if the game does not run at all, there is less damage done. Also, never trust anything you get back from a server.

Bram

[meds]

A lot of indie devs release alphas for free or even as a bonus to customers who pre-order, I think doing something like that is a good way to lower your stress because then you have a rough idea where your game is at in terms of bugs on a wide range of computers.

I've read the developer for minecraft has made one million euros on sales for his game even though it's still in alpha, maybe you won't make a million yourself releasing an alpha as a for sale thing but you might get some fnding to keep developing your game.

[flavio]

We use continuous integration: having a long testing time frame (and not a short period at the end of the project) allows to find many bugs with low efforts by testers. So you can have more testers (because the effort is low) and (I think) you get a very good analysis with this kind of tests. The problem is that it's hard to set up.

[HairyTroll]

I think if you do some basic research into hiring a proper QA studio based in India, you'd be absolutely stunned how cheap it is. I mean, seriously!

Do you provide tests scripts and the expected results? When you receive bug reports, are these sufficiently descriptive such that the bug can he recreated?

[Wrote A Game or Two]

I've read the developer for minecraft has made one million euros on sales for his game even though it's still in alpha

Yeah, but comeon, that's minecraft which as we all know is a complete anomaly in an otherwise barren wasteland where many indie developers work 80 hours a week for 3 months to make $200 in the apple app store.

[electronicStar]

Well from the Paypal debacle we have learned that he made something like $700k in the last month alone

[sake-bento]

As soon as I release a game, I do get nervous that there's a crippling bug or some endless loop that I completely missed. Thankfully, nothing horrendous has happened yet, but I've had enough bug reports post-release on nearly all my games.

I did get a lot of confused players for one of my games because it started within a meta-game and everyone thought they were supposed to play the meta-game instead of the actual game. That was a lesson learned. >.>

Thankfully, I have a pretty dedicated team of testers who abuse my game in terms of looking for bugs, and I've managed to find a few precious "non gamers" who are nice enough to at least go through some of my game so I can see if the gameplay is intuitive. I also release alphas to fans (who are dying to be on the inside track) and people who have provided good feedback on released games in the past.

[Jack Norton]

I've read the developer for minecraft has made one million euros on sales for his game even though it's still in alpha, maybe you won't make a million yourself releasing an alpha as a for sale thing but you might get some fnding to keep developing your game.

That's different: if you look in online games (webgames, MMO) is quite common to do open alpha/beta. Otherwise is impossible to test. But they put sales links already, so people who wants to contribute, can (a bit like preorders for normal downloadables).
Even nexic's MMO last time I checked was marked as "beta" but monetizes it already quite well

[Nexic]

Yeh, been in beta for almost 3 years now. Loathed to take down the sign since bugs are still popping up even now.

[Bad Sector]

That's different: if you look in online games (webgames, MMO) is quite common to do open alpha/beta. Otherwise is impossible to test. But they put sales links already, so people who wants to contribute, can (a bit like preorders for normal downloadables).

IIRC Minecraft was a SP-only game when Notch started asking for money.

I don't think alpha presales aren't going to work. It seems to work for Notch and for the Wolfire guys (for Overgrowth) which is also a SP-only game (from what i can see).

[vjvj]

Focus testing is a whole different beast compared to bug testing/general compat testing. I don't think it's a realistic requirement to expect one source to be able to cover both.

Primary requirements of bug testing are pretty simple; well-organized and thorough test plan, clear documentation output, good communication with test lead, and all the necessary hardware to cover the required range of end-user hardware. IMO this is something that's cost-effective to just outsource.

Focus testing is a lot trickier because it ties into your marketing. Who are your core players? Who are the fence-sitters and potential new fans? Only you can really answer that question, and someone is going to have to do the leg work to find them.

I also think it's important to stress the difference between "gamers" and the actual core players of your game. Even if you have an army of students/girlfriends nearby for testing (and let's assume they are all gamers), it doesn't necessarily mean you are getting good core testing. If your testing team consists of Starcraft players, Halo players, and World of Goo players, and your game is a Metroid title, you really aren't getting any core testing done. You'll get good feedback on interface, presentation, and pacing (i.e. "potential new fan" testing), but core testing needs to be done by people who understand and have experience with the genre.

[Applewood]

Do you provide tests scripts and the expected results? When you receive bug reports, are these sufficiently descriptive such that the bug can he recreated?

They had an overall brief of what to expect, but there'd no pedantry involved - they seem to test everything and report on gameplay crapness as well as obvious stuff like a crash when the user does x then y then z. Repro steps are usually pretty goodm but quality does vary and sometimes we send bugs back for more info.

Generally speaking, you can send your exe and a manual to these people and they'll do a thorough job - at least as thorough as you'd expect when paying a western company 10x the price.

I'm sure there are bad ones too, but again the same is doubtless true here as well. Bottom line, the cheap price isn't because you get a shit service, they just need less to thrive.

[ManuelMarino]

about Elemental, I'm very tempted to buy it... the fact that it's unfocused and complex... as a Master of Magic fan... who cares? Maybe I'll wait a couple of more patches and then buy it. Devs are trying to complete the game and solve anything can be solved with patching. I'm sure they'll be successful.