The 2006 2D Sprite Render Test

[dxgame]

Yes, I'm hoping for something REALLY valuable for reference in this thread.

TGB, PTK, BlitzMax, Flash, Custom Engines, etc, etc. Alot of 2D tools have been released the past year. I would like to propose that a new sprite render benchmark be outlined so we can see some real world tests of various rendering engines and how they perform on our own equipment. After the program structure is agreed on, hopefully a few versions of the test would be coded and made available for others to download.

Here's my basic program proposal we can start the discussion with:

Resolution: 800x600x32. Should be able to run in window or full screen.
Texture: A single texture used, 256x256 in size. (png or jpg)
Sprites: 32x32. Inside the texture 16 different sprites would be defined.

Program Design:

Windows Platform, DirectX (any version) Preferably a "2D via 3D" solution.

Vsync turned off so program can render as fast as possible.

Up to 2000 sprites will be rendered.

*Program starts off with 500 sprites.

Left arrow key removes 100 sprites.

Right arrow key adds 100 sprites, up to 2000 total.

* When a sprite is reset or initialized, it should have a random start up screen position. A random image from the texture. And a random alpha value.

Sprites move in a pong fashion around the screen, bouncing off the edges.

Sprites move 1 pixel per frame update. (No time based code, etc.)

Frame rate display should be toggled on/off by pressing the "F" key. When turned off, we can validate the frame rate by using something like Fraps, etc. Even though the Fraps I have only supports DX8 and up?

Number of sprites on displayed should be toggled on/off by pressing the "S" key.

ESC key exits program.

I would love to see how TGB stands up to PTK or BlitzMax, etc, etc.

Hopefully someone can design the basic texture to use.

And finally, I hope this thread does not die with no interest.

[ggambett]

Sounds cool, although it may not be representative of a real game. There are other variables and other limitations to consider. The texture and the sprites are conveniently power-of-2, for example. I have code to deal with arbitrary shaped textures and arbitrary shaped images, which has a little performance penalty.

The only big variable you left out is windowed vs fullscreen.

[Phil Steinmeyer]

Are the sprites opaque (alpha 255), transparent (alpha 255 or 0), or translucent (alpha 0, 1, ... 254, 255)?

[Indiepath]

I think that any system should be out of the box without any 3rd party modules or improvements. If not then the results could be way off depending on the code used.

For example, here are some results of my Single Surface module for BMAX, the speed increase is compared against the standard product.

DirectX - 40% Less Video Ram 28% speed increase
OpenGL - 42% Less Video Ram 28% speed increase AND 57% faster than DirectX

[undersan]

The biggest problem, IMO:
With 32x32 sprites, these apps will likely be bound by the sheer number of pixels involved (fill-rate). The per-sprite cost--things like setting render states and setting up vertices--will be completely hidden. Why is this bad? Because, for any hardware-accelerated renderer, it's the per-sprite processing that really impacts the CPU, and most games are CPU-bound, not GPU-bound.

One possible solution:
How about buttons to double/halve the sprite size, from tiny (1x1 pixels) up to something absurdly large, say, 256x256? With larger sprites, you'll see the renderer's per-pixel efficiency (fill-rate). With smaller sprites, you'll see its per-sprite efficiency ("sprite-rate" ).

One more thing:
Don't cap the number of sprites at 2000. I'd again suggest buttons to double/halve the sprite count, from 1 up to something absurdly large, say, 1M.

I might be willing to put up a DX8 version (w/ source) at some point. Indiepath is right, though--the biggest value of this benchmark will be comparing straight-out-of-the-box commercial engines.

[Mike Wiering]

I think it would be more interesting to not have any requirements like (only Windows, must use DirectX, only use 1 single 256x256 texture, out-of-the-box functions, etc.), but just say what the program must do (display 500 moving sprites, etc.) and let people implement this in the best possible way (as long as you provide the source code). It might be interesting to see how OpenGL compares with DirectX, and Windows vs. Mac / Linux.

What exactly are you planning to measure? The number of FPS for 500 sprites? I think it would be more interesting to measure max. number of sprites possible in 100 FPS for example. Then there should't be a limit of 2000 sprites.

I'ld split this into:
- test 1: transparent sprites, no alpha blending
- test 2: translucent sprites with random alpha (if supported)
so you can compare this with DirectDraw / GDI / older 2D libs as well. When choosing a system/library for a game, you'll probably have decided wether or not you're planning to use alpha blending.

Some libraries might have very slow text functions (since you'ld normally use bitmap fonts instead of text), so I'ld suggest to only display the last FPS value for a moment when you press F (counted without any text on the screen), or use bitmap fonts and display it real-time.

[dxgame]

hmm.. the more complicated it gets, the more likely no one will ever code something up. I still think a simple sprite render test using 32x32 sprites with random alpha sprites moving around via a pong style would be good enough. I also think any method is relevant, so if you have your own mod to an engine or whatever, let's see that as well. Source to me is not manditory. I'm sure if someone writes a demo that is performing ALOT faster than everyone else's inquiries will have to be made.

I guess allowing more then 2000 sprites is fine, but probably would only matter on higher end computers.

[Pyabo]

I think this is a great idea...

Coincidentally enough, I've already written most of this test to run under PTK, when I was evaluating it. I think the sprite I used is only 16x16, but they bounce around the screen pong-style... and I can adjust the number of sprites and change between fullscreen and windowed on the fly.

Dxgame, since this is your idea... maybe you should write a template C++ program to run the test. Just leave some nice comment blocks where any library-specific code needs to go.

[Indiepath]

You might also want to test with a couple'a hundred thousand sprites since GPU tech nowadays is going to cope with pretty much anything you chuck at it.

[dxgame]

couple'a hundred thousand sprites? jeesh.. I pretty much run only on older gpu/cpu systems with onboard graphics capabilities.

"Dxgame, since this is your idea... maybe you should write a template C++ program..."

I'm a VB6 person, sorry. I am going to throw together a demo to get things rolling. I'm sure my engine will probably be the slowest, because it's so high level, but oh well. Should have something ready later today.

[electronicStar]

I have written a prototype of 2D engine that fills exactly these conditions (complete with the pong-like collisions) in JAVA (using JOGL for OpenGL).
I'll see if I have enought time to clean it up and upload it.
The 2000 sprites barrier should be no problem I seem to remember testing with up to 3000 alphaed sprites on an AMD 2600+ with GF ti4200.

[PaulModz]

This is a great idea, but here's the rub; the harder you try to compare oranges to oranges, the further the stats drift from real-world performance.

I think the most doable way to approach it would be to start with a reference implementation that everyone can copy and a set of assets that everyone can share. I'm currently moving my GDI+ vector graphics code to 2.5D using Direct3D, and I'd be happy to write a managed (C#/.NET) version.

Don't use random positions, images or alpha values. The position, texture, alpha, etc. of each sprite should be identical each time the program is run, like 3DMark, with no user user input during the benchmark. You could put all the relevant sprite info into an XML or comma-delimted file to be sure everyone is using the same values.

It would also be helpful to have the benchmark run a series of tests in multiple resolutions and windowing modes.

[electronicStar]

This is a great idea, but here's the rub; the harder you try to compare oranges to oranges, the further the stats drift from real-world performance.

I think the most doable way to approach it would be to start with a reference implementation that everyone can copy and a set of assets that everyone can share. I'm currently moving my GDI+ vector graphics code to 2.5D using Direct3D, and I'd be happy to write a managed (C#/.NET) version.

Don't use random positions, images or alpha values. The position, texture, alpha, etc. of each sprite should be identical each time the program is run, like 3DMark, with no user user input during the benchmark. You could put all the relevant sprite info into an XML or comma-delimted file to be sure everyone is using the same values.

It would also be helpful to have the benchmark run a series of tests in multiple resolutions and windowing modes.

I think the original idea was just to see different implementations in action, what would be the point to just copy and compile the same thing for everybody?

[Mike Wiering]

Don't use random positions, images or alpha values. The position, texture, alpha, etc. of each sprite should be identical each time the program is run, like 3DMark, with no user user input during the benchmark. You could put all the relevant sprite info into an XML or comma-delimted file to be sure everyone is using the same values.

Or use a very very simple random generator. That's alot easier than reading values from a file. Something like:

Global Seed = 0
Function Random (X)
  Seed = (101 * Seed + (Seed Shr 5) + 5659) And $FFFFFF
  Return Seed Mod X
End Function

[dxgame]

Looks like it's hard for developers to agree on the simple things sometimes. In any event, here's a little demo I wipped up that shows what I was talking about:

http://www.dxgame.com/download/SpriteTest.zip (400k)
Requires Windows XP, DirectX8, VB6 runtimes (already on XP systems)

F: Toggle Framerate Display
S: Toggle Sprite count display
Right Arrow: Add 100 sprites (up to 2000)
Left Arrow: Remove 100 sprites
R: Init sprites with rotation
ESC: Exit

Each time the sprites are initialized, a random sprite image is defined from the texture, a random alpha value, a random direction to Pong around the screen with.

I threw the rotation option in just to emphasize a real usage scenario because I'm always rotating sprites in my games.

I already know my engine is slower than others, but I'm mainly interested to see how TGB, PTK, Blitz, etc, perform in a similar test.

[undersan]

http://www.ericology.com/temp/SpriteMark_1_0.zip (537k)

This is DX8.1 and C++. I wanted to see if I could reach my Radeon 9600 Pro's advertised peak fill-rate of 1.6 gigapixels/sec, so I went kinda overboard with the render-state options. I got up to 1.3, meh, pretty close.

Note to benchmarkers: Even with the full display turned off, my app's framerate is still displayed top-right (only every 6th frame, to save time!)

I haven't compatibility-tested my D3D framework, so I apologize in advance if this crashes on certain cards.

As dxgame predicted, his VB6 version is slower than my C++ version. Here is the comparison on my system (2000 32x32 sprites, windowed):
dxgame: (deleted, see my later post for correct results)
undersan: (deleted, see my later post for correct results)
Update: My initial testing was slightly flawed. I had the DirextX debug runtime libraries enabled--d'oh!

[dxgame]

Hi Undersan! Thanks for coding your example.

"As dxgame predicted, his VB6 version is slower than my C++ version. Here is the comparison on my system (2000 32x32 sprites, windowed):
dxgame: 56 fps
undersan: 190 fps"

hmm... this is a pretty big difference so I ran some tests on my system(s) to compare yours, especially ingteresting since they are both using DX8!

My results were radically different than yours:

On my Intel Extreme Graphics Card my VB6 engine actually performed better than your C++ version, in both windowed and full screen with 2000 sprites.

dxgame: 40fps
undersan: 39fps

I then tested it on my Raiden Express 200 Series computer, results were slightly faster but both engines were VERY close again:
undersan: 69fps
dxgame: 65fps

THIS is the reason these kind of tests are great. For some reason my engine is performing poorly on your Radeon 9600 Pro compared to your engine. Perhaps there is a hardware render state I should be setting when running on a 9600 pro?

Thanks again, lets hope some others chime in. This could be very interesting!

[Mike Wiering]

Still, I think displaying the text influences the results too much in dxgame's program: with 100 sprites, I get about 320 fps if the "100" is displayed and 455 if not, but that is with fps display on. The moment I press F, it shows 550 or more for a second.

[dxgame]

"Still, I think displaying the text influences the results too much.."

If I was using bitmapped fonts the performance hit would have been alot less, just a few more sprites then. In this example I'm using the standard DX8 Text Rendering which is extremely slow and should never be used for anything serious in my opinion. I suggest using a 3rd party fps displayer like Fraps to help verify frame rates.

[Indiepath]

Here's the BlitzMax (DX7) Version : http://www.indiepath.com/tim/SpriteTest.zip

T: Toggle FullScreen/Window.
F: Toggle Framerate Display
S: Toggle Sprite count display
Right Arrow: Add 100 sprites (the sky is the limit)
Left Arrow: Remove 100 sprites
R: Toggle Rotation
ESC: Exit

#################

Here are my results for 2000 Sprites Windowed

- BlitzMax : 196FPS
- DX Game : 94FPS
- Undersan :108FPS

Here are my results for 2000 Sprites FullScreen

- BlitzMax : 230FPS
- DX Game : 102FPS
- Undersan : 119FPS

FYI, I can run 10,000 Sprites in BMAX at 60FPS.

[dxgame]

Thanks IndiePath for the BlitzMax version!

hmm... I'm seeing a disturbing trend. We seem to be getting better performance on our own machines compared to running on other machines. IndiePath stated speeds better than DOUBLE my engine on his machine, yet on my machine the BlitzMax version runs slower than the 2 previous demos. The attached screen grab shows it running at 36fps.

"FYI, I can run 10,000 Sprites in BMAX at 60FPS." I got 10,000 at 10fps on my box.

[Indiepath]

Results from another test machine

BMAX: 40FPS
DXGame: 35FPS
Undersan: 23FPS

[dxgame]

More Results: (This is my highest end machine that I dare develop on.)
AMD Sempron 3100+ 1.8ghz
Geforce 4 MX 4000

2000 Sprites Full Screen

BMax: 76fps
DXGame: 68fps
Undersan: 46fps

[Indiepath]

I think I know why my initial results were very high. BMAX will use TnL if it's available, if not then it falls back to the standard HAL renderer (thanks Tim). The card on this machine supports TnL, my second test machine does not.

[jankoM]

hmm... I'm seeing a disturbing trend.

It seems your machines like your engines the most

undersan I think you have a bug , when It says 4 sprites I see 3, when it says 8 I think I see 6, when 2 - 1 and when 2000 I see 1978 sprites

ps. joking about the last one

[Mike Wiering]

undersan I think you have a bug , when It says 4 sprites I see 3...

Maybe there is one running around with alpha 0?

[dxgame]

"TnL" Ok, I'm thick on this one, what's TnL?

[Indiepath]

"TnL" Ok, I'm thick on this one, what's TnL?

Hardware transformations and lighting.

[electronicStar]

Still, I think displaying the text influences the results too much in dxgame's program: with 100 sprites, I get about 320 fps if the "100" is displayed and 455 if not, but that is with fps display on. The moment I press F, it shows 550 or more for a second.

I might very well be the consequence of a bug (eg fps not measured when writing is OFF for example)

[Hidden Sanctum]

I tried various numbers:

At 2,000 sprites:
Blitz - 506
DX -103
Undersan - 423

At 1,000 sprites:
Blitz - 885
DX -227
Undersan (1,024 sprites) - 620

At 500 sprites:
Blitz - 1430
DX -346
Undersan (512 sprites) - 680

At 100 sprites:
Blitz - 2750
DX -603
Undersan (128 sprites) - 720

At 10,000 sprites on the Blitz demo, I got 104fps.

My system specs are 3.2 Ghz Pentium (dual CPUs), 4 Gig ram, 256M nVidia geForce 6800.