Testing and Quality Assurance service. Soon!

[VaderSB]

Hi guys,

we're about a couple of weeks from starting to offer compatibility and performance testing and QA (Quality assurance) services. We already built a diversified computer park with a broad range of hardware from Pentium3 to PentiumD, from RivaTNT to Radeon X800. We're going to target Windows based games, testing will be available from Win98 to Vista.

Currently we're preparing final specifications, gathering statistics and organizing the service. As we really want to meet your expectations right from the start we decided to ask you - please tell us your thoughts, ideas, recommendations on such a service. What do you want it to include? Any special case tests (We've already prepared some)? How much would you pay for such service?

About us:
We've been in indie gaming scene since 2000. Our game site is www.vsbgames.com. In 2006 we started to prepare our next project: www.vsblabs.com. It's already working and we're already offering design services, but we delay the public announce until Testing and QA Service release.

[cliffski]

Compatibility testing is cool, but I'm more inteersted in good old-fashioned bug testing.
When a customer finds a bug, they can't be as helpful as someone who understands debugging, and testing procedures etc. Also, you aren't paying them, so they won't put much time into it.
Ideally I'd like a service where I can send a debug build, and get a report back that illustrated:

Any bugs that were found, and accurate steps to reproduce those bugs
Any play imbalances or exploits that were discovered
Any performance concerns on specific systems

An accurate breakdown of what the minimum spec for the game seems to be
A list of hardware configurations the game was tried on, and any notes about performance on those systems

How much would I pay for it? I don't know! But I think you need to be able to scale things accross the whole spectrum. I think most people here would happily pay $99 for a quick and simple verification of no obvious show stopper bugs, and a test accross a few hardware platforms. Other people would probably happily pay a few hundred dollars, or more for a more in-depth report.
The best thing would be to see a sample report for a previous client, together with how much it cost :D

[VaderSB]

Thank you Cliffski! Actually this is what we bet on - we have deep understanding how gaming software works and will be able to provide developer with much more detailed and useful reports.

Along with gameplay imbalances/exploits we'll work to find graphical glitches that will help to apply final polish.

[ZuluBoy]

Excellent idea!!!!
Please go ahead and launch it. I will probably be your first client :D .

[VaderSB]

Excellent idea!!!!
Please go ahead and launch it. I will probably be your first client :D .

Cool! :) I'll PM you when the service will be launched! We're about a week from public release.

[Nexic]

I would also be very interested in this, please PM the details to me when you launch. Note that adding Mac testing would probably result in a dramatic increase in clients. As an example I have lots of friends and family to test windows platforms but don't know anyone with a mac to test on.

Of course I'm buying my own mac, but being able to test several hardware configs would be nice.

[fmsoftware]

I'd be very interested in getting a professional bug report of my software.
Also I would like a real world video capture of the game running on slower PCs to make sure it's playable.
Different sound card configuration can also be a problem I'd like testing.

[VaderSB]

Thank you for your feedback!

I would also be very interested in this, please PM the details to me when you launch. Note that adding Mac testing would probably result in a dramatic increase in clients. As an example I have lots of friends and family to test windows platforms but don't know anyone with a mac to test on.

Of course I'm buying my own mac, but being able to test several hardware configs would be nice.

We were thinking about Mac testing but we'll not include it in initial start. Maybe we'll add it later if we'll see enough demand.





I'd be very interested in getting a professional bug report of my software.
Also I would like a real world video capture of the game running on slower PCs to make sure it's playable.
Different sound card configuration can also be a problem I'd like testing.

Video capture will not be provided by default, but we're working on it. Hopefully we'll provide it on-demand. We've also got several different sound cards to test on (including rare/old ones) along with detailed stats generated with DxCapsViewer and DxDiag.

[luggage]

Video capture will not be provided by default, but we're working on it. Hopefully we'll provide it on-demand. We've also got several different sound cards to test on (including rare/old ones) along with detailed stats generated with DxCapsViewer and DxDiag.You should really supply video capture by default, not the opposite. All the testing I've done in a commercial environment involved recording the session onto VHS (that was how long ago it was), and for any bugs to note down the time on the tape. It was incredibly useful as it allowed coders to rewind back as far as they needed to to see what the tester actually did.

If you supplied a capture of the data it would also give your customers safe knowledge that you have actually tested the game for the specified time. For example, if I asked for 6 hours testing, I'd get 6 hours of footage to watch through if I needed to.

Otherwise there'll always be a nagging doubt that the total testing time has been 40 minutes in a mad dash to hopefully get as many bugs as possible and look like the job was thorough.

[cliffski]

I'm not fussed about video capture of the screen. In an ideal world I'd want a split-screen video capture of the game, and someones face as they played it.
That would be really cool. (some top notch testing companies do this).

[lennard]

This is a topic that is near and dear to my heart at the moment because my last couple of projects have been slow to release because, even though I wasn't finding bugs, it wasn't clear to me that the game was "done" and so I kind of meandered.

Now, if I could pay a hundred bucks and feel secure... I'd probably do it. But since $100, even at $5 an hour, only gets me 20 hours... From big dev. I'm used to a small army of $9 an hour kids playing the games in shifts, 24/7 and serious bug reports every morning. We just can't afford that kind of overhead in the indie. space (OK, I can't) and something short of that isn't going to make me feel all warm and fuzzy and get me out the door quicker (something worth some real dollar amount).

No real answers, would be happy to hear other peoples strategies but, unfortunately, I suspect that my real answer is that I'm going to be beating my next game to death as well unless I can trick somebody else into doing it.

BTW, I sometimes pay the kids a buck or two per reproducible bug... handy to have kids if they will do this kind of thing for you and maybe this is actually my best MO.

[VaderSB]

Thank you all for such detailed feedback!

I would like to tell some technical details we've developed. Our service will include two 'sub-services':
1)Compatibility and Performance testing (will be measured in cycles). Each cycle will include running the provided game on about 20 configurations across 12 computers for 10 minutes per each configuration. The goal of this testing is to find out if game runs/crushes on given configuration, what the performance (fps) and if there are any hardware-related glitches.
2)Quality Assurance testing (will be measured in hours of gameplay). This will include finding software-related bugs, gameplay glitches, graphical glitches, usability problems and so on.

Now about the video capture question. There are 2 different possible reasons for it (as Luggage already stated): as a proof of testing was done completely and as a bug-report tool. Let's talk about these reasons separately.

1) (Proof of work). I don't really believe that it makes big sense. There would be tens and hundreds and thousands of video footage, the overhead is too big.
So, instead of video, the key points of our business strategy are: 1)Reports quality 2)Word of mouth marketing (positive clients feedback is very important) and 3)Money Back guarantee (Client can get full refund if he'll be dissatisfied with our reports quality). Along with our reports client will be provided with all logs and computers detailed statistics. Actually the amount of information provided by default will be pretty large, and it won't be a problem to ask us for some special info or testing conditions.

2) (Video as bug-report tool). Well, that makes more sense. It's very possible if we'll use it right from the start - currently we plan to use it for Quality Assurance testing sub-service on one video per reproducible bug basis. I think it will be better to use some kind of video capture program like FRAPS.


@Lennard,
I understand your doubts, but anyway you can give our service a try. After all if you'll not be satisfied with first reports you'll be able to just get a full refund, so it will be safe for you.

[TunaBreeze]

Sounds like the start of a great service although I would highly recommend adding video recording. The recordings don't have to be at full resolution, as long as the client gets enough information to figure out what might be the cause of a bug. And having a recording of the full session isn't always necessary, you could offer full recordings or partial recordings a few minutes before a bug is spotted. This would give you more payment plans and I think more customers. And hard drive space is very cheap these days, you can get 1TB of storage for about $300 these days. Would be a great benefit to your service.

Are you by chance hiring in home testers?

[VaderSB]

@TunaBreeze
Thank you for feedback! :)
As for the video recording we're already ready to provide our clients with it, it's just experimental now, that's why we don't mention it on the site.

And no, we don't hire home testers. That's one of our key features - testing is done by people with real game development experience, not just by gamers. :cool:

[TunaBreeze]

<<< Experienced programmer (and gamer), and aspiring game developer. Good to hear your offering video services.

[JoshuaSmyth]

Sounds like a good service idea. I'd probably be more interested in performance and compatibility testing, than bug tracking and QA. Oh, and I'd be interested in Mac testing. But my next products are a long way off, so no need to go out and buy a bunch of Macs just for me :)

[VaderSB]

<<< Experienced programmer (and gamer), and aspiring game developer. Good to hear your offering video services.
Although at this point we don't need more testers, but we'll keep you in mind. Thank you!

Sounds like a good service idea. I'd probably be more interested in performance and compatibility testing, than bug tracking and QA. Oh, and I'd be interested in Mac testing. But my next products are a long way off, so no need to go out and buy a bunch of Macs just for me :)
Thanks for feedback! Well, we're planning adding Linux and Mac testing in future but we concentrate on windows-based games testing for now.