Dont you get scared at release?

Discussion in 'Indie Related Chat' started by zoombapup, Sep 5, 2010.

  1. zoombapup

    Moderator Original Member

    Joined:
    Nov 25, 2004
    Messages:
    2,890
    Likes Received:
    0
    I was reading about the failure of Elemental at release over at rockpapershotgun

    http://www.rockpapershotgun.com/2010/09/03/blindness-caused-elemental-release-fail/

    It made me think of something I hadnt really considered much yet. That is, the release testing!

    It scares me silly that I'd release something that literally couldnt be played by anyone but me. Although I like to think I've got a fairly pragmatic design sense at least. I've got some fair resources (in the form of my students) when it comes to basic testing, but as anyone whose released a major game will tell you, complex games have a LOT of bugs. Hell we used to have a pretty big test departnment and still shipped with bugs. Now its just me I reckon I'll have kittens when trying to release my game.

    How do you guys handle testing cycles? Do you have alpha/beta tests? Do you have a significant number of testers? Do you find you have lots of bugs on release? If so, how do you handle the fallout of those bugs?

    I've always thought that having a "soft launch" would be the way to go, but I've heard plenty of people now saying that isnt such a great idea. So maybe instead it should just go into a long beta with a fairly limited set of testers instead?

    Thoughts please?
     
  2. Desktop Gaming

    Moderator Original Member Indie Author

    Joined:
    Feb 24, 2005
    Messages:
    2,296
    Likes Received:
    12
    I didn't really do much alpha testing for Magicville - only two people other than me saw it before Beta. At Beta I had I think seven people plus me testing and they found a lot of stuff - the game was played from start to finish twice over by one person, plus the other testers trying to break it, before I was happy to let it go. It was in beta for about six weeks.

    When I released Magicville it had a doozy of a bug in it, caused by third party code.

    The code I used attempted to set permissions for the "Users" group... the code referenced this group by name, not its constant. This turned out to be very, very bad, as the group isn't called "Users" on any non-English OS, causing the whole game to fail.

    How did I miss that? Well, I'm English and have an English OS! I didn't see it coming in a million years but come, it did. I got a revised build out within a day but it still took a few days to filter through to market. My conversion rate took a nasty hit (the bug-free version's conversion rate was 3x that of the first release). I don't like to think about how much money it cost me in lost sales.

    How will I stop it happening again? It probably won't, now that I know! But even so, I have the German Edition of WinXP running on my Mac via VMWare.

    When people find post-release bugs I try to be apologetic and positive in getting it quickly fixed. People usually understand!
     
  3. Applewood

    Moderator Original Member Indie Author

    Joined:
    Jul 29, 2004
    Messages:
    3,859
    Likes Received:
    2
    I think if you do some basic research into hiring a proper QA studio based in India, you'd be absolutely stunned how cheap it is. I mean, seriously!
     
  4. cyodine

    Original Member

    Joined:
    Jun 22, 2005
    Messages:
    169
    Likes Received:
    0
    A friend of a friend also swears by the effectiveness of India's tech support.

    As for QA, I'm wondering if India's cheap QA is more of a test against bugs, or whether they have companies with experienced gamers that can give sound feedback on the gameplay as well. Does anyone know?

    And speaking of exploiting / enriching (however you want to view it) the third world, anyone know much about Samasource? I think it's IT services based in Africa.

    As for release bombs, I guess it does kind of scare me a bit, although only slightly since there's no social expectation for my first release to be great; if it bombs it'll probably just be viewed as the status quo. I won't have tons of negative publicity; I'll have just none.

    I guess frequent tester feedback throughout the development process is essential to avoid "blindness". I'm wondering if many of you privately network with those you trust in this forum to get sound outside feedback on your game's progress without the much greater risk of having your ideas snatched (current separate forum thread) via just exposing all your work to the general public.
     
  5. Applewood

    Moderator Original Member Indie Author

    Joined:
    Jul 29, 2004
    Messages:
    3,859
    Likes Received:
    2
    My involvement has been to have a game we wrote for someone else passed through an Indian company, and we just had feedback about the usual bugs and stuff.

    I was impressed by how thorough it all was and they caught a variety of things that wouldn't show up by just going through the motions. Repro steps were clear and in gud Inglund.

    Apparently the bill was 500 quid and that included the complete pass of bugs and then a full regression test a week later when we'd allegedly fixed them all.

    Regarding getting general gameplay feedback, some might off that but I'm not interested tbh. It's better to trial your stuff on a small group of the intended audience and you can usually get that done for the price of half a dozen pizzas.

    Regarding "abuse" of the 3rd world, that's not really something I have a problem with and don't consider it to be abuse. Technically, India is in the 3rd world as it's not a part of Nato or the Warsaw Pact, but it's a pretty solid country with good people and a democratic government and that's all I need to know.

    From my perspective, the only reason not to outsource to foreignors revolves around depleting the jobs market at home but that's something I'll spend more time worrying about once my own company is in the fortune 500 tbh.
     
    #5 Applewood, Sep 5, 2010
    Last edited: Sep 5, 2010
  6. princec

    Indie Author

    Joined:
    Jul 27, 2004
    Messages:
    4,873
    Likes Received:
    0
    We have a whole room full of Indians here at my current employed. Suffice to say I don't think they're worth the electricity they use.

    Here at Puppy Towers I spend approximately 50% of game development time testing. It's a relatively expensive way to do it but as I'm also the producer I'm fairly sure that I get what I want :)

    Then we tend to release betas quite early to pick up the technical troubles. We also log all exceptions and failures automatically and every point release I sort them in order of incidence rate and fix the top 10. This has done wonders for quality control.

    I would like to find a cheaper mechanism for testing than unit testing or using my time. Unit testing is hailed in the business world as some sort of mantra but my empirical observations is that unit testing effectively doubles development time and catches very, very few bugs.

    Cas :)
     
  7. Applewood

    Moderator Original Member Indie Author

    Joined:
    Jul 29, 2004
    Messages:
    3,859
    Likes Received:
    2
    Total agreement on unit testing - just the latest emperors new clothes.

    You shouldn't do your own gameplay testing though, seriously. You get so used to what you're looking at and checking that what you did actually does, that the bigger picture is easily missed and often even forgotten.

    You'd probably never notice there isn't a way to quit the game properly because you left in the <esc> bugs out immediately hack for example.

    Regarding your room-full of Indians, there's nothing inherently brilliant about your average Indian citizen. I recommend them purely because they're phenomenally cheap when they're in India! :)
     
  8. ChrisP

    Indie Author

    Joined:
    Feb 5, 2007
    Messages:
    971
    Likes Received:
    0
    Funny story: During much of Mayhem Intergalactic's development, there was a crash bug that occurred with 100% reliability whenever you started a game, quit out to the menu, and then started another game. Pretty easy-to-find bug, right?

    It went unnoticed for months. Because every single time I tested the game, I would start a new game, play it for a while, and then quit the entire app to make changes and recompile before starting a new game. I was so narrowly focused on testing the "actual game" that the idea of testing other sequences of menu button-presses completely escaped me. When I did blunder into the bug one day, it was completely by accident.

    These days I try to be a little more comprehensive, but you can never be sure that you've found everything yourself. So getting as many eyes as possible on the product before release is an absolute must, however you choose to do it.

    I can echo DG's comment about non-English versions of Windows, that tripped me up as well. Though in my case it was broken Unicode handling in some file loading code (passing a UTF-8 string into the ASCII version of a Windows function... doh).

    Oh, and then there was the time I updated Steam achievement progress every time you performed the game's core action... which triggered popups that built up, spammed the screen and neatly obscured vital parts of the UI for minutes at a time. Meaning the Steam version of the game was essentially unplayable, despite extensive pre-release testing. You see, those popups don't actually appear until the game is publicly available via Valve's servers. Good times.

    So yeah, I'd say release is pretty scary. :)
     
  9. princec

    Indie Author

    Joined:
    Jul 27, 2004
    Messages:
    4,873
    Likes Received:
    0
    Yes this is probably S*ny's mistake - importing them over here and having to pay 10x the amount from them. Most of which goes to the p1mps.

    Doing my own gameplay testing is definitely a bit of a bummer but I put on a player hat most of the time when I'm doing it. Unfortunately means I get sucked into a playtest session for an hour. Sometimes I test things out on the girlfriend :)

    Cas :)
     
  10. Dan MacDonald

    Moderator Original Member Indie Author

    Joined:
    Jul 26, 2004
    Messages:
    1,424
    Likes Received:
    0
    I'd love to understand more what went wrong with Elemental. It sounded more like the game mechanics were broken then technical issues. I've always admired Brad Wardell and I do appreciate his honesty in criticizing his own company. I really didn't know what to do with his justification.

    It sounds legitimate enough, at the same time this is the reality of developing indie games for most indie developers. In most cases one individual is doing all the programming and design and able to maintain objectivity and deliver quality, the competent ones anyway. So I have to disagree with his justification. I think there are some cases where his statement is true, but it's not true of most successful indie developers I know.
     
  11. zoombapup

    Moderator Original Member

    Joined:
    Nov 25, 2004
    Messages:
    2,890
    Likes Received:
    0
    Its definitely a trap that many can fall into though Dan. I mean its always brutal when you put a game you think is finished in front of real players and they just flounder around looking confused.

    I've seen grown men almost in tears when faced with a player that literally cannot understand what to do. It seems so obvious when you are developing the thing, but in the cold light of someone else's experience things just arent percieved the same.

    I guess I have to hope that my Uni ties will see me through a half decent round of testing at least. I suggest that others do the same (find a local university with a game dev course and approach them to do some testing for you, they will be surprisingly happy I would suspect).

    Plus for the most part students are often genuinely wanting to help improve the product too.
     
  12. Bram

    Indie Author Greenlit

    Joined:
    May 1, 2010
    Messages:
    179
    Likes Received:
    2
    Paranoia is sometimes good, and in-app-purchases may help

    When it comes to release, I get really paranoid, and check, check and check again. This saved me from most big screw ups.

    The biggest screw ups so far for me were:

    (1) I released a game which I had tested on iPad/iPhone but not iPodTouch. The latter crashed with black screen. I only had .xib files for former platforms.

    (2) I wrote my own leaderboard code, which managed to poison all clients: When one user used an exceptionally long gamername, it got stored by my leaderboard server, however the server truncated the string halfway through a unicode.
    When the server then sends to all users: all users would crash: my game was no longer playable for anyone with internet connection.

    The first bug was really mitigated by the fact that it was a free game with in app purchases: this means that not a single paying customer was affected!

    So my advice: make game free, use in-app-purchases: if the game does not run at all, there is less damage done. Also, never trust anything you get back from a server.

    Bram
     
  13. meds

    meds New Member

    Joined:
    Dec 26, 2008
    Messages:
    344
    Likes Received:
    0
    A lot of indie devs release alphas for free or even as a bonus to customers who pre-order, I think doing something like that is a good way to lower your stress because then you have a rough idea where your game is at in terms of bugs on a wide range of computers.

    I've read the developer for minecraft has made one million euros on sales for his game even though it's still in alpha, maybe you won't make a million yourself releasing an alpha as a for sale thing but you might get some fnding to keep developing your game.
     
  14. flavio

    flavio New Member

    Joined:
    Jan 13, 2008
    Messages:
    104
    Likes Received:
    0
    We use continuous integration: having a long testing time frame (and not a short period at the end of the project) allows to find many bugs with low efforts by testers. So you can have more testers (because the effort is low) and (I think) you get a very good analysis with this kind of tests. The problem is that it's hard to set up.
     
  15. HairyTroll

    Original Member

    Joined:
    Jul 29, 2004
    Messages:
    582
    Likes Received:
    0
    Do you provide tests scripts and the expected results? When you receive bug reports, are these sufficiently descriptive such that the bug can he recreated?
     
  16. Wrote A Game or Two

    Wrote A Game or Two New Member

    Joined:
    Oct 9, 2009
    Messages:
    179
    Likes Received:
    0
    Yeah, but comeon, that's minecraft which as we all know is a complete anomaly in an otherwise barren wasteland where many indie developers work 80 hours a week for 3 months to make $200 in the apple app store. :D
     
  17. electronicStar

    Original Member

    Joined:
    Feb 28, 2005
    Messages:
    2,068
    Likes Received:
    0
    Well from the Paypal debacle we have learned that he made something like $700k in the last month alone:eek:
     
  18. sake-bento

    sake-bento New Member

    Joined:
    Jul 23, 2009
    Messages:
    23
    Likes Received:
    0
    As soon as I release a game, I do get nervous that there's a crippling bug or some endless loop that I completely missed. Thankfully, nothing horrendous has happened yet, but I've had enough bug reports post-release on nearly all my games.

    I did get a lot of confused players for one of my games because it started within a meta-game and everyone thought they were supposed to play the meta-game instead of the actual game. That was a lesson learned. >.>

    Thankfully, I have a pretty dedicated team of testers who abuse my game in terms of looking for bugs, and I've managed to find a few precious "non gamers" who are nice enough to at least go through some of my game so I can see if the gameplay is intuitive. I also release alphas to fans (who are dying to be on the inside track) and people who have provided good feedback on released games in the past.
     
  19. Jack Norton

    Indie Author

    Joined:
    Jul 28, 2004
    Messages:
    5,130
    Likes Received:
    0
    That's different: if you look in online games (webgames, MMO) is quite common to do open alpha/beta. Otherwise is impossible to test. But they put sales links already, so people who wants to contribute, can (a bit like preorders for normal downloadables).
    Even nexic's MMO last time I checked was marked as "beta" but monetizes it already quite well ;)
     
  20. Nexic

    Indie Author

    Joined:
    Nov 5, 2004
    Messages:
    2,437
    Likes Received:
    0
    Yeh, been in beta for almost 3 years now. Loathed to take down the sign since bugs are still popping up even now.
     

Share This Page

  • About Indie Gamer

    When the original Dexterity Forums closed in 2004, Indie Gamer was born and a diverse community has grown out of a passion for creating great games. Here you will find over 10 years of in-depth discussion on game design, the business of game development, and marketing/sales. Indie Gamer also provides a friendly place to meet up with other Developers, Artists, Composers and Writers.
  • Buy us a beer!

    Indie Gamer is delicately held together by a single poor bastard who thankfully gets help from various community volunteers. If you frequent this site or have found value in something you've learned here, help keep the site running by donating a few dollars (for beer of course)!

    Sure, I'll Buy You a Beer