Launching a crowdsourced beta testing service?

Discussion in 'Indie Business' started by MattInglot, Oct 3, 2010.

  1. MattInglot

    Moderator Original Member

    Joined:
    Jul 26, 2004
    Messages:
    170
    Likes Received:
    0
    Once upon a time (2003ish?) I had come up with the idea of a crowdsourced beta testing service for indies called BetaShare. It actually came to fruition and had a few successful betas run through it (!), but with another business requiring my 100% attention I made the difficult decision to shutdown what was looking like a promising experiment.

    I am seriously considering rebuilding and relaunching this service, and I am looking for feedback on the appeal of this idea:

    The main point of the service is to connect indie developers with people who wish to test games/software, and facilitate a way to do so while also attracting publicity for their project. Key features would include:

    1) Developers sign-up and receive a testing space for their in-beta game/app, including key functionality as bug reports, feature suggestions, uploading and managing multiple releases, mass-e-mails to testers, surveys, etc. Focus would be on providing high quality core toolset.

    2) Beta testers are attracted through a combination of advertising the beta to existing subscribers looking for new betas to try, our own social media presence, and the developer inviting their own user base.

    3) In exchange for the service of testing, beta testers receive a free license of the complete product. Developers have the ability to offer additional incentives above and beyond this, but at minimum your testers (deservedly) get the final product free. To avoid freeloaders, testers would have to earn a certain amount of feedback points.

    4) In the spirit of crowdsourced development, features for capturing what the crowd really wants would exist, such as being able to up/down vote feature requests/bugs and survey the audience.

    5) Betas can be public (hit a button and congrats your a tester) or restricted to approved testers (ie you apply to be a tester).

    6) A web API would exist to allow developers to easily integrate bug submit functionality in-game. Initially we would provide something easy to communicate with + code samples for common languages.

    7) Considering finding ways to help the product launch post-beta.

    I have several freemium based revenue model ideas. The base service would be 100% free, which might entitle you to all core functionality, a certain tester cap, and so on. Inviting your own testers to become part of the service would likely carry some sort of rewards (a key challenge will be building this audience that makes the service so valuable).

    Looking for Feedback on...

    I'm interested in any and all feedback on this idea, but I am particularly interested in the following questions:

    1) Would you use such a service? Why or why not?
    2) Is finding sufficient beta users something that is currently a challenge?
    3) What WOULD make you use (or not use) the service?

    Feedback not along the lines of the above is also appreciated. HONESTY is really important. I would rather this thread go down in flames than to invest time/money into a project that won't appeal to its market. :D
     
  2. andrew

    andrew New Member

    Joined:
    Jan 14, 2007
    Messages:
    487
    Likes Received:
    0
    I would absolutely pay for something like that.

    In my ideal world, you could say "I want 1500 minutes of testing", and decide whether you want 100 testers for 15 minutes or 10 testers for 3 hours. Would be nice if the testers could also give you directed feedback as to fun factor, etc, and could make some level of feature requests/suggestions.

    I do agree that the trick will be in keeping enough high quality testers around to make it worthwhile for the developers. For example FGL has their "first impressions" service (more for fun factor/reviews than actual bug hunting) but the the quality of the testers is very spotty, and in some cases many of the reports are plain unusable.

    - andrew
     
  3. Olofson

    Original Member

    Joined:
    Dec 11, 2005
    Messages:
    270
    Likes Received:
    0
    Definitely interesting!

    Maybe a rating system for reports would help...? That should separate the serious testers from those who just want to get paid for playing games, I believe.
     
  4. Hosebomber

    Hosebomber New Member

    Joined:
    Jul 28, 2010
    Messages:
    3
    Likes Received:
    0
    You could also use this as a service for those wanting to break into the gaming industry as testers. Those who noticeably put time and effort into testing and giving valid feedback could have a social page like linked in. Developers could leave feedback on the testers page if they liked the testing they did.
     
  5. Jasmine

    Jasmine New Member

    Joined:
    Aug 8, 2010
    Messages:
    195
    Likes Received:
    0
    This is a nice idea. I'm wondering what feedback would be like from so many though?

    ...

    Perhaps testers could have a forum where they could discuss their thoughts on what they played, and perhaps a system of voting up/down other people's opinions.

    The opinions with the most ups are presented as the major points of the report. Less popular options could be listed later in the report as opinions from individual testers.

    Also, the game could be rated by all players with stock questions:

    eg,
    Did you enjoy playing the game -- 3/5
    Rate difficulty (1=too easy 3 = perfect 5 = too hard) -- 4/5

    Then make bar graphs.
    Code:
    Did you enjoy playing the game?
    5/5 -- 8 players -- 12%
    4/5 -- 16 players -- 25%
    3/5 -- 24 players -- 37%
    2/5 -- 7 players -- 10%
    1/5 -- 3 players -- 4%
    
    Then profile the testers on factors like age and gender, and see if the "did you enjoy playing the game" shows bias to any age one group or gender.

    Code:
    Did you enjoy playing the game?
    Male -- 35 testers -- average 3.41/5
    Female -- 30 testers -- average 2.26/5
    
    Under 15 -- 7 testers -- average 2.46/5
    16-25 -- 11 testers -- average 3.10/5
    26-35 -- 8 testers -- average 3.04/5
    36-45 -- 3 testers -- average 2.62/5
    46-65 -- 2 testers -- average 3.50/5
    66+ -- 1 tester -- average 2.0/5
    
     
  6. Vino

    Vino New Member

    Joined:
    May 16, 2010
    Messages:
    345
    Likes Received:
    0
    I have a questionnaire for offsite testers that I might use for this service, but most of my valuable playtesting comes from onsite testers that I can watch play the game directly. I receive probably 90% of usable feedback from playtesters that I can meet face to face and physically watch playing my game. Your service would be most valuable if it could provide a screencap video of the player testing the game, or if it could connect me with local playtesters. What are the chances of this?
     
  7. barrygamer

    Original Member

    Joined:
    Apr 10, 2006
    Messages:
    490
    Likes Received:
    0
    Random thoughts-

    I've always been interested in a service which checks that an app (or a feature in it) works *at all* on a variety of hardware. e.g. all the iphone models.

    I'm paranoid, so I'd be a little wary about who I was giving my product to. I guess devs could give restricted versions.

    I guess you've seen it but Gamezebo lets users sign up to be beta testers and give feedback to devs. Haven't looked in detail what they do. Obviously, very casual game oriented.
     
  8. Nutter2000

    Original Member Indie Author

    Joined:
    Jul 27, 2004
    Messages:
    993
    Likes Received:
    3
    Just to expand on Barry's idea a bit, it might be a bit technical for an indie service but, some sort of tracking system like EA's would be very useful.

    e.g. logging levels reached, when the app runs, when the app shuts down successfully, when it doesn't, and so on.

    Having that as well as a survey
     
  9. Vino

    Vino New Member

    Joined:
    May 16, 2010
    Messages:
    345
    Likes Received:
    0
    Oh yeah, if it sends me back a minidump after it crashes that would be pretty pro.
     
  10. Nutter2000

    Original Member Indie Author

    Joined:
    Jul 27, 2004
    Messages:
    993
    Likes Received:
    3
    It's the kind of Premium service I would probably pay for
     
  11. speeder

    speeder New Member

    Joined:
    Aug 22, 2010
    Messages:
    237
    Likes Received:
    0
    Wow, that is exactly what I am looking for (actually, I signed in the forum to make a post in the feedback forum... unfortunately I could not, I kept getting messages that as new user I could only post in the feedback forum, exactly what I wanted... :/)

    I would not pay for it, but that is because I don't have money (I think I am actually bankrupt by anticipation... since I have 0 money, and loans to pay next year...)

    But I would use it a lot :)

    And yes, finding testers is hard, REAAAALY hard, my game got 3 months added to schedule, because it took me 3 months to get meaningful feedback (people would play the game, and then say it is nice, and go away... :/ I had to offend some people to make them try to offend me back by saying what on the game sucked... something that actually worked... it allowed me to make a huge list of defects, including obvious stuff that I missed, like the fact that I forgot the "skip" button in cut-scenes...)

    In fact, I planned to finish my game, and then market it. After I noticed how I was not finding testers, I decided to do the inverse, hype it, use the user feedback and then complete it. Unfortnately neither is working properly (because not marketing it results in no testers, and marketing it results in lots of effort to have very few testers, mostly because my game is too work in progress for people tastes...)
     
  12. TimS

    Original Member

    Joined:
    Feb 9, 2005
    Messages:
    686
    Likes Received:
    0
    Some things that would help (from my limited perspective)...

    1) - If a game is non-indie (published), the 'free copy' thing might not be so easy to put together. I'm not sure how else it would work, but if there was a way to get a 'free copy' to say... Big Fish players... that'd be good.

    2) - I agree 100% with Vino in that having video of the players actually PLAYING the game would be very, very nice. This was super helpful to us, and we weren't even doing anything weird. I imagine it'd be a million percent awesome for anyone trying to do something a bit more experimental.

    3) - A focus on player types. Developing casual Puzzle Adventure games, it would be worth less than zero dollars to me to have 1000 15-year-old male flash gamers play the game and give feedback. If you could devise a system to split the groups of players into reasonable categories so that each developer could target their beta to the audience that matters to them, it would help a lot.

    As for your final three questions:

    Sure I'd use it, if it was functional for what I'm doing at the moment (casual). If I ever leave casual for more uncharted waters, I'd use it all the more. Currently we don't have any trouble finding beta people, because we're dealing with BFG (they send out a survey build).

    Now if you can combine this with a crowdsourced QA service... you'll really have my attention! Like Barrygamer said, (but not for iphone), the massive world of PC gamers (and mac) different system configurations and hardware is a major headache. I would pay for a service that would just send out an exe that say... plays a video with a certain codec... if I could get back fps numbers and dxdiag files (and crash reports). :)

    -T
     
  13. Chris England

    Chris England New Member

    Joined:
    Sep 22, 2009
    Messages:
    81
    Likes Received:
    0
    Having flicked through the thread, I think two things struck me as really good ideas:

    1) The video of people playing the game. What people identify as a problem is not always what the problem actually is; having independent verification of what they are doing would be amazingly helpful.

    2) Giving the ability for developers to leave feedback on testers. That would encourage people to try and join up as testers and do a genuinely good job, as it'd help build their resume.

    Whether I'd actually pay for it, I don't know. The advantage people get of pre-ordering on my site is beta access, so hopefully I'll have enough beta testers I wouldn't need to (and I'd be monetising beta access, rather than paying for it). However, if I were using a different business model, I'd definitely be interested.
     
  14. Qitsune

    Qitsune New Member

    Joined:
    Mar 19, 2007
    Messages:
    358
    Likes Received:
    0
    Well, it could double as a service such a model mayhem which links models to photographers and artists, so you could find testers in your area. And maybe you could even use those that got high ratings by testing online.
     
  15. MattInglot

    Moderator Original Member

    Joined:
    Jul 26, 2004
    Messages:
    170
    Likes Received:
    0
    Wow thanks for all the great feedback everyone. Some really interesting points brought up here and neat ideas like kickstarter.com

    It honestly never occurred to me to have the ability to capture live videos of people playing. To be honest, I also really doubt we'd be able to pull it off as a "version 1" feature, which leads me immediately to two questions for the people who are behind it:

    1) Beyond this feature (which would be super cool), would the service still be useful as a means of connecting developers to beta testers and providing nice tools for people to report bugs, submit feature requests, rate each other's requests, etc?

    2) Does anyone have some ideas on how capturing video of gameplay could be accomplished? My immediate thought is that the sheer diverse nature of platforms and development systems for games out there may make this extremely challenging at best. My second thought is that someone out there is likely be able to prove me wrong.

    It's never too late to start collecting interested names. I don't yet have a newsletter sign-up, but if you wish to be e-mailed about any future developments on this project, care to help provide deeper feedback, just want to be the first to test drive this thing if it does get built, please feel free to PM me.
     
  16. Nutter2000

    Original Member Indie Author

    Joined:
    Jul 27, 2004
    Messages:
    993
    Likes Received:
    3
    1) yes I'd say it would

    2) You could get them to install something like CamStudio for video capture
    Although I haven't used it myself so you'd have to research it to make sure it all works ok but I have heard good things.
     
  17. speeder

    speeder New Member

    Joined:
    Aug 22, 2010
    Messages:
    237
    Likes Received:
    0
    Video capture is tricky unfortunately :/

    My game for example, I found only one single way to make it work properly:

    Use CamStudio lossless codec.
    Use other capture software other than CamStudio (CamStudio captures the window borders too... :/), for that I am using VirtualDub
    Use a ramdrive driver, so I can record to the RAM. (my disk is not fast enough to write the lossless data that my game requires...)

    The ramdrive in special requires lots of fiddling, it is finnicky, it sometimes crash the entire OS... I guess that forcing volunteers to install that is not good idea.
     

Share This Page

  • About Indie Gamer

    When the original Dexterity Forums closed in 2004, Indie Gamer was born and a diverse community has grown out of a passion for creating great games. Here you will find over 10 years of in-depth discussion on game design, the business of game development, and marketing/sales. Indie Gamer also provides a friendly place to meet up with other Developers, Artists, Composers and Writers.
  • Buy us a beer!

    Indie Gamer is delicately held together by a single poor bastard who thankfully gets help from various community volunteers. If you frequent this site or have found value in something you've learned here, help keep the site running by donating a few dollars (for beer of course)!

    Sure, I'll Buy You a Beer