Unorthodox Beta Testing

[soniCron]

Since Jeweltopia's development has already taken such an unorthodox approach, I figured I'd carry that on to the beta testing process. The first beta was pre-occupied with technical feats (does it run?, are there bugs?). The second beta solidified the core gameplay (is it fun?, do you enjoy it?). The upcoming third (and final) beta will resolve and focus on the presentational issues. The reason I chose to split the beta into three distinctly different versions was calculated and intentional; I wanted to filter user responses into increasingly valuable categories:
The first beta strongly relied on a mass audience for discovering technical errors -- the greater the number of testers, the more accurate the results.
The second beta utilized players who had been intrigued by the gameplay of the first beta -- only fans of the inital concept stuck with it, and thus their opinion became more valuable, since they have increased interest in the core mechanics.
The third and final beta will narrow the audience to the select few that truly enjoy the game, and as such, their opinion on the presentation is the most valuable. Since they are the core players/fans, their input on the presentation should prove to be the most insightful and useful. What a random passerby thinks of the presentation on the first iteration is unimportant, since they most likely will not play the game in the future.

It is well known that with the greater the number of iterations of betas, the fewer testers will respond. This is beneficial because it helps to weed out useless commentary ("Put in DRAGONS! With lasers from their eyes!") and it distills the audience into increasingly insightful positions, who's opinions are progressively more valuable.

There are certainly many reasons not to follow this approach. A largely disillusioned audience was the cause of such unfocused testing with the first beta, for example. This can lead to a great number of people that may not play the game in its final iteration, and thus may lose a few eyes and/or sales. (In retrospect, this would have been most successful if done with a privately controlled audience, with the post-presentational beta being tested publicly. Of course, I had no such audience to draw from, so that was not an option.)

However, I did not see this as a significant loss simply because fellow developers aren't my target audience. As always, with the good comes the bad. Nevertheless, I feel the benefits of such an approach to beta testing are vastly more valuable because it results in a more tuned and focused product.

This isn't the only way to beta test and focus your product, but I know it helped me weed out useless and unhelpful suggestions. When inundated with dozens of responses, it becomes increasingly difficult to judge the value of each opinion. By distilling the testers through this natural progression, I was able to achieve a greater signal to noise ratio than I would have if I had used a more traditional finish-everything-then-test beta testing procedure and I was able to more accurately hone the features of the beta to appeal more strongly to the most appreciative players. This, in turn, results in a more focused game that should prove to be more valuable to my target audience.

You can't please everyone, so please the ones you can the most.

What are your thoughts or insight to this type of testing? What beta testing methods do you employ?

[Graeme]

It makes sense to target your audience for the third beta as you've stated. But here's a question: How are you going to find those people and then conivince them to test your game?

[soniCron]

How are you going to find those people and then conivince them to test your game? I'm not entirely sure what you mean. If you're talking about this current public beta, then they are users from the previous versions. If you're talking about a private beta, then I would include friends, family, fellow developers, and willing newsletter members. This should procure enough interest to use the funneling method in an effective manner.

[Graeme]

You said that you wanted participants of the third beta to be people that you're actually attempting to sell the game to (or at least the people you are most likely to sell it to), but friends, family and fellow developers are likely to have a different opinion than the actual target audience. For example, other developers will always have a different perspective and may not be the kind of people that play match-three games anyways. Friends/family may work, if you happen to know that they are the match-three kind of crowd, but if not, they would also have a different perspective in my opinion. Not only are they outside the target audience, but the fact that they know you may affect their input. If you really want to do the third beta as you described, it seems to me that you'd want to find random people that like casual games. People that you don't know, who don't also develop games. However, those people could be hard to find. Hence, my original question: How will you find them?

[Graeme]

After re-reading your post I think I misread it the first time, you are going to use people from the previous beta that liked the game for the third beta, yes? I thought you were going to use people that tend to like that kind of game, not people that actually did like your game. That makes sense. Good approach ;)

[soniCron]

That is correct. Thank you! :)

[Chris Evans]

Sounds good, but I don't know how "Unorthodox" this process really is. :) Seems pretty standard to me.

I will say though, in the initial public beta release not only do you want to pay attention to technical errors from the testers/players, but also common design/ui/pacing frustrations that they encounter. For example, if 3-4 people make remarks that they have trouble starting a game because of the UI, then you know there's a problem with the UI design. Likewise if 4-5 people are unclear of level objectives/goals, then it's obvious you'll need to address it. So I wouldn't say the initial beta phase is just for finding technical errors, but it's for also finding common themes and complaints. If 3-4 people complain about something, there's a good chance several hundred other people feel the same way.

Another thing, I'm not sure I'd let your "core" testers determine your presentation. Presentation is usually something you first notice and sets the tone of the game. But after awhile, you become accustom it. I personally prefer to get presentation feedback from brand new users since I want to make a first good impression even with the "passer-byers" you spoke of. With "core" testers, I like to focus on usability and any outstanding technical issues. Core testers/players are great for usability because they can tell you if something becomes "annoying" or "boring" after repeated plays. For example, you might think it's okay to have a quick 5-second cut-scene between levels. A core tester may tell you that even though it's only 5 seconds, he wants to be able to skip the cut-scene. They may also request customizable keys and such.

Though like I said, the process you stated for the most part seems pretty standard. You'll notice Nexic did something similar with his Mighty Rodent game as did Cas with Titans. Also some people have private beta tests with their existing customers or dev mailing lists before they even post their beta on this board. In fact, I know very few Indies (who aren't working with a publisher) who don't use a similar process. But you will notice with every new game your betas that you release will tend to be more complete because ideally you'll be working off of a proven/tested code-base from the start. You'll have a better idea of what works and what doesn't. So it may seem that some of the experienced developers here are releasing betas of near-complete games but it's actually because they're using their established code-base and UI designs and have also conducted private tests with customers and other devs.

[soniCron]

Sounds good, but I don't know how "Unorthodox" this process really is... it may seem that some of the experienced developers here are releasing betas of near-complete games but it's actually because they're using their established code-base and UI designs and have also conducted private tests with customers and other devs. Thank you, Chris, that was very perceptive. :) My only real world experiences with beta testing were with large-scale applications and games, so as you can imagine, in comparison, the two methods of testing are vastly different in many ways. In retrospect, I see exactly how Nexic, Cas, etc. are taking the same approach. It seems much of the indie game community (and the shareware community as well?) are very progressive in this regard! It's great to have these ideas validated by other, more seasoned veterans of development.

However, I disagree that the broadest audience should be used for presentational (not UI) issues. (By the way, excellent perception on the UI testing!) What's the point in attracting someone who doesn't even like the game? I'd rather have my players appreciate the environment more than a general audience. A popular opinion may draw more eyes, but are they as valuable?

[Monkeyget]

When i saw your post i immediatly sought of that tip from joel on software :
http://www.joelonsoftware.com/articles/BetaTest.html

11. Most beta testers will try out the program when they first get it, and then lose interest. They are not going to be interested in retesting it every time you drop them another build unless they really start using the program every day, which is unlikely for most people. Therefore, stagger the releases. Split your beta population into four groups and each new release, add another group that gets the software, so there are new beta testers for each milestone.

Notice that some other tips don't apply to indie games.

[Tom Gilleland]

I think it is quite interesting that you are working toward an optimum beta process. I enjoy reading your posts, they are always brutally accurate, or brutally creative, or sometimes just brutal! ;) But I think your analysis of why testers look at each version only equates to a smallish subset of the testers. I personally only looked at your first version, wrote up my impressions, then went back to work. My limiting factor was available time, not interest in the product or other reasons.

Tom