Do you use TDD?

[MikeIG]

I've just happened to stumble upon this article and I'm curious, what do you think about Test Driven Development?

Usually I'm TDD-ing only some bits of code, tried once (ok, many times ) to develop all the app this way but after one point I had to change my tests so much that I let it go.

What about you? Especially Flash developers can you use TDD with AS? (I have no experience in flash programming)

[Bad Sector]

Its possible (although you'll probably need some external tool, like Flex).

However TDD (and unit tests in general) have this issue of that if you want to do it properly, the testing code becoming larger than the code it tests. Also there is the extra issue of writing tests which subconsciously avoid some of the edgy parts. To avoid this you need to split the coders in those who will write the tested code and those who will test the code based on the public API.

Given that unit testing usually makes sense in teams of programmers (especially large-ish teams) and that to make it properly and with so-called blackbox testing (the above method i mentioned) to avoid the case of writing code that avoids the edgy parts you need even more people, its probably not very useful in the realm if indieness :-).

Also personally i don't see the point in writing so many tests for games at all. Its not banking software.

Now i'm sure somebody out there does this, but i'm also sure that the majority of (indie and not) games developers do not and the gain isn't that big for games.

[JarkkoL]

I don't do TDD, but I have written fair amount of unit tests for lower level code (matrix/vector/container/etc. classes) because they may cause most havoc when misbehaving and are easiest to test. Even when there is plenty of test code to write to have good test coverage, the code is very straightforward and usually faster to write (when comparing line to line at least) than the code you test with it. I don't test only functionality of components either, but have code in place for performance tests and also some empirical multi-threaded code tests. I must admit though that I'm a bit lazy in writing tests and usually just have random burts to complement the unit tests suite and writing them is a good place to waste some time if you are facing a coder's block (:

When you refactor some of the code it's been pretty useful too to keep the code working. Just yesterday I refactored some skeletal animation playback code (removed some code duplication) and had few unit tests written for some of the low-level components, which triggered because a small mistake I had made at refactoring (forgot to normalize quaternion for generic interpolation code that I refactored to be shared for interpolating translation & rotation animations). Without the unit test the animation would have just behaved strangely and it would have been much more difficult to narrow down where the problem is. Infact, I wouldn't probably even realized there was a bug caused be the refactoring because the animation system isn't ready and thus isn't generating meaningful result yet (:

I also use fair amount of templates in code and you need concrete code which instantiates that template code or compiler doesn't catch even if you write garbage within those functions, and unit tests serve good purpose there too.

[princec]

TDD is most helpful in larger teams all working on different bits of things. If you're the only programmer, it's a big ol' waste of time.

It also doesn't help much in game design terms, as game designs tend to change a lot more rapidly than even the requirements for more mundane programming projects. As soon as some fundamental bit of design changes in a game it usually means a big refactor and bang! - all the tests are broken and need to be rewritten.

Also... there is this idea that writing the tests first is somehow big and clever. It's not. What is big and clever is designing the application first, but the TDD-tards of this world seem to think that writing unit tests somehow compensates for total lack of design.

Cas

[jamesm]

I read this article a while ago: http://gamesfromwithin.com/backwards...en-development

Found it interesting, but I dont use TDD

James

[dewitters]

TDD is most helpful in larger teams all working on different bits of things. If you're the only programmer, it's a big ol' waste of time.

You're correct that it's more useful in large teams than with lone wolves, but I don't agree that it's a total waste of time. I think it's best to find the right balance between absolute TDD and no tests at all.

I use TDD for code that goes into my 'libraries'. TDD forces you to design your code as modular as possible, and you can test it straight away before writing a single line of actual game code. So it speeds up my work right at the beginning, and makes sure I don't take lazy shortcuts and end up with spaghetti code .

Remark that I use python with py.test, which doesn't require the boilerplate code that other test frameworks need.

[Cygon]

I'm not doing full TDD - I'm quite often writing my code first and then the tests. But I'm using unit testing quite a lot.

It has caught many problems in code that I thought I had designed to perfection.

It also forces me to write classes in a way that allows them to be isolated from the rest of the code (so I don't have to initialize half the game just to be able to construct an instance of the class).

I simply don't have much trust in un-(unit-)tested code anymore. When I design something complex, I always fear that I have overlooked something, like an object entering an unexpected state or a series of calls causing a code path to be entered in a way I didn't foresee.

By measuring test coverage (which lines of code are exercised by my unit tests), I know that at least all possible code paths have been taken once.

I can't, however, support the typical argument by the TDD crows that the time invested in TDD pays off. It does lead to an increase in code quality, both by reducing bugs and by improving design, but it in no way equals the time invested. As long as I have a day job, I take the liberty of designing a solid framework for my future games, but as I full-time indie, I would write tests very selectively only in parts of the code where I believe there's a danger of messing up

My game has 1,599 test cases at the moment and a test coverage of 92%.

[Bad Sector]

The idea for most people is to finish that damn game ASAP so they can leave said day job :-P

[JarkkoL]

Added a test counter out of curiosity and currently got 5,422 tests total which are testing decent portion of the components in two of the core libraries of my engine. I probably got poor test coverage though d: This constitutes ~13% (~15kloc) of all the code in the engine, but probably only couple of % of all the coding effort, which I'm sure is already saved back as the result of more reliable code base. What tools do you use to measure the coverage?

[Cygon]

I'm working with .NET, so I can use the excellent NCover together with NUnit. NCoverExplorer generates nice html reports, so I can publish these on my build server (TeamCity) and even fail the build if coverage goes too low.

Its only drawback is that NCover requires a Windows system, so I can only run my unit tests but not measure their coverage on my Linux box.

[James C. Smith]

TDD is great

I would argue that the tests themselves are not the most important characteristic of test driven development. TDD is all about the repetition of a very short development cycle. It’s about having a program that works from day one (but does very little) and continues to increase in functionality as you go. In many ways it is the opposite of writing lots of design documents, designing object models in UML or making other diagrams, architecting big systems and coding for days on something that can’t run yet. If you avoid all that, and just start with the smallest amount of code that can work and then you build incrementally and refractor as needed you will be doing something very similar to TDD even if you never write a single test.

[Acord]

Now, if you're talking straight TDD type stuff, where tests are created first to feed data in and data out before any other code is done, it works great. For big projects. It keeps you from wasting a lot of time when someone's module doesn't work.

For small stuff, the research indicates that you are better off either following an iterative design process with a few testers or cowboy coding it.

So - if you have a very modular design AND more than two or three coders, it could be beneficial. Otherwise, it is probably not going to help you unless you want to be able to put it on a resume'.

[Cygon]

Most advocates of TDD and agile practices in general tend to state that it works best for small to medium sized projects (Google Search: Agile in Large Projects).

So - if you have a very modular design AND more than two or three coders, it could be beneficial. Otherwise, it is probably not going to help you unless you want to be able to put it on a resume'.

I'd say a modular design is the result of TDD. Since unit tests force you to design classes in a way so they can be used on their own (and don't depend on ten others to initialize). You'll often even hear that Unit Testing is more about design than about finding bugs.

I can vouch that it certainly changed the way I design my classes - I tended to create components as black boxes: all classes within a component could be tightly coupled with each other as long as the component's public interface pedantically fulfilled its contract. Now I modularize with a much finer granularity and I have to say that this causes me far fewer headaches.

[Acord]

It could be. I've heard of it being drastically mis-used before by groups that are not prone to plan well in the first place. I have to plan - I am just absolutely no good at cowboy coding.

[princec]

Nah, modular design is the result of 30 years coding experience. After a while you just write things this way because it's what tends to work best.

Actually modular is too broad a specification. I'd say it was more client/server mostly.

Cas

[Cygon]

Sure, experience is everything. Programming is all about tacit knowledge and you don't get that with a meager 5 years of writing code or so.

The "modularization" I picked up on and believe Acord was talking about is on a much lower level than chopping up a project into client and server, so to say

What fascinated me was that most agile projects I've looked at could be decomposed and reused at the class-level. Simply because when I write a unit test for the class "HighscoreDisplay", I don't want to have to initialize DirectX, load the score texture or let the "HighscoreDisplay" class go shopping for references to other objects.

Thus a unit testing developer designs it so it gets its references handed to it as interfaces in its constructor. Then the unit test can create mockup implementations of those interfaces to test the class in isolation. And when I want to reuse it in my next game, I don't have to dig it out from ten other classes it was interacting with.

[Nutter2000]

Yes but that sort of modularisation has been going on longer than TDD with MVC and many other such design patterns over the years.

I would say TDD encourages modularisation, but it's not the direct result of it.

TDD is like a lot of things, especially in coding, in that it's no use to anyone unless they understand what went before and why people thought it was needed in the first place.

Use it right and TDD can be a valuable asset in making sure vital code works with various input, but use it wrong results in adding massive unnecessary bloat to your development time, especially for the lone wolf developer.

Really? HighscoreDisplay? Assuming that's a class to display your highscore, what exactly are you effectively testing without providing it with the ability to display anything?
I would say that's better tested with functionality testing in situ rather than a unit test.

[JarkkoL]

If you got a generic HighscoreDisplay component, it makes sense to have mock object for graphics interface and test that calls to the interface are as expected with given input data to HighscoreDisplay component.

[Nutter2000]

Yes, except that in that case you'll have a generic gfx interface, so most of the potential bugs are most likely to be in the generic gfx interface to actual graphics interface rather than your highscore class so in cases like this these bugs are best tracked in a complete functionality / beta test which you should be doing as well as unit tests.

My point is that if you have the time to do it and, preferably, someone else paying the bills then by all means unit test everything but I would suggest that most developers don't have that luxury especially indie developers.

[JarkkoL]

Yes, except that in that case you'll have a generic gfx interface, so most of the potential bugs are most likely to be in the generic gfx interface to actual graphics interface rather than your highscore class so in cases like this these bugs are best tracked in a complete functionality / beta test which you should be doing as well as unit tests.

My point is that if you have the time to do it and, preferably, someone else paying the bills then by all means unit test everything but I would suggest that most developers don't have that luxury especially indie developers.

Yes, you would need some kind of generic gfx interface, but graphics interface implementation isn't really any more prone to errors than any other part of the code. That claim strikes me odd and in fact, generic graphics interface is in much more heavier use than a highscore class, thus goes through far more testing. Anyway, I don't think you have the luxury of not do any unit tests

[Nutter2000]

Well what I mean is that by definition it is more complicated because it's dealing with potentially several actual interfaces once implemented and is most likely a heavier class therefore more prone to bugs.

heh, don't get me wrong, I'm not saying TDD or unit testing in general is a bad thing, I think it's a very powerful tool. I've done many projects with and without it, both commercially and otherwise.

However, what I've found from my experience is that it does take a lot of time, especially if you're a one man team then you need to at least double your development time probably more to take into account writing the test cases.
Quite often writing unit tests takes longer than writing the actual code and it still doesn't catch everything.

So my point is that if you have the time and money available to spend the extra time doing it then go for it, otherwise, and this is true for most projects, look at your coding smart, decide where your bugs are most likely going to occur and unit test just those parts while leaving the rest for functionality testing.

I'm just not a fan of jumping on to the new coding principles as if they're second coming and everyone else must convert or die as a heretic!!!!
Not saying anyone here does that but you do get it a lot in coding.

Just trying to point out that TDD is like any other coding principle from the Waterfall Methodology to Scrum and beyond, they all have their good points and bad points so it's best to know what these are and know when to use them or not.

[Cygon]

@Nutter2000: You reaction seems a bit like you were responding to some language war.

I believe none of posters here ever said TDD is the best thing in the world or that TDD can turn a neophyte into a badass developer or tried to pressure others to adopt it.

Yes but that sort of modularisation has been going on longer than TDD with MVC and many other such design patterns over the years.

I would say TDD encourages modularisation, but it's not the direct result of it.

No complaint about the first part. Modularization has been a buzzword before OOP came around.

All I'm trying to communicate is a particular thing I've observed: that unit tests, because you test very small pieces of functionality, force you to make those small pieces of functionality isolatable from the rest of the code. And I put that effect in the broader category of modularization.

They not only encourage, but enforce that particular brand of modularization.

It's still the programmer's job to know what to turn into separate units, where to set up interfaces and how best to abstract things - that's where the OOP concepts and design patterns apply. But if the programmer gets it wrong, he'll notice the tightly coupled mess he created when he tries to write a test for it.

Really? HighscoreDisplay? Assuming that's a class to display your highscore, what exactly are you effectively testing without providing it with the ability to display anything?
I would say that's better tested with functionality testing in situ rather than a unit test.

That it does the right calls to - let me come up with some name - your IHudDrawer interface. Just as much as required to test that any possible error cannot be in the HighscoreDisplay class.

And then you have another test that makes sure the DrawString() method of your actual IHudDrawer implementation generates the right vertices to put the characters from your font texture onto the screen.

The in situ test you talked about also has its place in the TDD world. It's called an "integration test" and done complimentary to unit tests.

[Nutter2000]

@Nutter2000: You reaction seems a bit like you were responding to some language war.

I believe none of posters here ever said TDD is the best thing in the world or that TDD can turn a neophyte into a badass developer or tried to pressure others to adopt it.

Sorry Cygon, that wasn't my intention

No complaint about the first part. Modularization has been a buzzword before OOP came around.

All I'm trying to communicate is a particular thing I've observed: that unit tests, because you test very small pieces of functionality, force you to make those small pieces of functionality isolatable from the rest of the code. And I put that effect in the broader category of modularization.

They not only encourage, but enforce that particular brand of modularization.

It's still the programmer's job to know what to turn into separate units, where to set up interfaces and how best to abstract things - that's where the OOP concepts and design patterns apply. But if the programmer gets it wrong, he'll notice the tightly coupled mess he created when he tries to write a test for it.

Yup, and I agree with everything you just said
I was just trying to inject a "Your Mileage May Vary" into it

[JarkkoL]

However, what I've found from my experience is that it does take a lot of time, especially if you're a one man team then you need to at least double your development time probably more to take into account writing the test cases.

How much time you invest in writing your test cases is totally up to you of course and not enforced upon you by some unit testing manifesto (: I mentioned earlier that I have probably put only couple of percent of my total development time writing the test cases even though they contribute fair amount of lines to the code base, and in my opinion if you put more than half on it you got your balance wrong (TDD zealots may and are free to disagree though ) I just like to try to wisely focus my efforts where it makes the biggest difference, so instead of just blindly testing everything I have so far focused my testing efforts to easily testable low-level code.

I feel that this investment of couple of precent of development time has huge pay off in increasing the code stability and thus I have to spend less time debugging more complex code, and I don't think I write more buggy code than the average programmer (: Let say if my matrix class would be less stable, I would spend tons more time debugging some more complicated functions dealing with matrix math, because the focus is in getting the math right and assuming these components just work, I find that removing extra complexity from debugging is hugely benefical. When I write some of the components, I want to test that those components in isolation anyway, thus adding the tests to the unit test suite is just a natural thing to do.

[Nutter2000]

That's pretty much the point I was trying to make

[CymnChutt]

I use FF and agree its slower these days. I have no notion of browser loyalty, so if you go for Opera Id be interested to know how it works out.