The 2006 2D Sprite Render Test

Discussion in 'Game Development (Technical)' started by dxgame, Jun 23, 2006.

  1. dxgame

    dxgame
    Expand Collapse
    Original Member

    Joined:
    Mar 11, 2005
    Messages:
    267
    Likes Received:
    0
    Yes, I'm hoping for something REALLY valuable for reference in this thread.

    TGB, PTK, BlitzMax, Flash, Custom Engines, etc, etc. Alot of 2D tools have been released the past year. I would like to propose that a new sprite render benchmark be outlined so we can see some real world tests of various rendering engines and how they perform on our own equipment. After the program structure is agreed on, hopefully a few versions of the test would be coded and made available for others to download.

    Here's my basic program proposal we can start the discussion with:

    Resolution: 800x600x32. Should be able to run in window or full screen.
    Texture: A single texture used, 256x256 in size. (png or jpg)
    Sprites: 32x32. Inside the texture 16 different sprites would be defined.

    Program Design:

    Windows Platform, DirectX (any version) Preferably a "2D via 3D" solution.

    Vsync turned off so program can render as fast as possible.

    Up to 2000 sprites will be rendered.

    *Program starts off with 500 sprites.

    Left arrow key removes 100 sprites.

    Right arrow key adds 100 sprites, up to 2000 total.

    * When a sprite is reset or initialized, it should have a random start up screen position. A random image from the texture. And a random alpha value.

    Sprites move in a pong fashion around the screen, bouncing off the edges.

    Sprites move 1 pixel per frame update. (No time based code, etc.)

    Frame rate display should be toggled on/off by pressing the "F" key. When turned off, we can validate the frame rate by using something like Fraps, etc. Even though the Fraps I have only supports DX8 and up?

    Number of sprites on displayed should be toggled on/off by pressing the "S" key.

    ESC key exits program.

    I would love to see how TGB stands up to PTK or BlitzMax, etc, etc.

    Hopefully someone can design the basic texture to use.

    And finally, I hope this thread does not die with no interest. :eek:
     
  2. ggambett

    ggambett
    Expand Collapse
    Moderator Original Member Indie Author

    Joined:
    Jul 26, 2004
    Messages:
    1,977
    Likes Received:
    3
    Sounds cool, although it may not be representative of a real game. There are other variables and other limitations to consider. The texture and the sprites are conveniently power-of-2, for example. I have code to deal with arbitrary shaped textures and arbitrary shaped images, which has a little performance penalty.

    The only big variable you left out is windowed vs fullscreen.
     
  3. Phil Steinmeyer

    Phil Steinmeyer
    Expand Collapse
    Original Member

    Joined:
    Mar 1, 2005
    Messages:
    757
    Likes Received:
    0
    Are the sprites opaque (alpha 255), transparent (alpha 255 or 0), or translucent (alpha 0, 1, ... 254, 255)?
     
  4. Indiepath

    Indiepath
    Expand Collapse
    New Member

    Joined:
    Aug 22, 2004
    Messages:
    999
    Likes Received:
    0
    I think that any system should be out of the box without any 3rd party modules or improvements. If not then the results could be way off depending on the code used.

    For example, here are some results of my Single Surface module for BMAX, the speed increase is compared against the standard product.

    DirectX - 40% Less Video Ram 28% speed increase
    OpenGL - 42% Less Video Ram 28% speed increase AND 57% faster than DirectX
     
  5. undersan

    undersan
    Expand Collapse
    Original Member

    Joined:
    Oct 25, 2004
    Messages:
    52
    Likes Received:
    0
    The biggest problem, IMO:
    With 32x32 sprites, these apps will likely be bound by the sheer number of pixels involved (fill-rate). The per-sprite cost--things like setting render states and setting up vertices--will be completely hidden. Why is this bad? Because, for any hardware-accelerated renderer, it's the per-sprite processing that really impacts the CPU, and most games are CPU-bound, not GPU-bound.

    One possible solution:
    How about buttons to double/halve the sprite size, from tiny (1x1 pixels) up to something absurdly large, say, 256x256? With larger sprites, you'll see the renderer's per-pixel efficiency (fill-rate). With smaller sprites, you'll see its per-sprite efficiency ("sprite-rate" :)).

    One more thing:
    Don't cap the number of sprites at 2000. I'd again suggest buttons to double/halve the sprite count, from 1 up to something absurdly large, say, 1M.

    I might be willing to put up a DX8 version (w/ source) at some point. Indiepath is right, though--the biggest value of this benchmark will be comparing straight-out-of-the-box commercial engines.
     
  6. Mike Wiering

    Mike Wiering
    Expand Collapse
    Original Member

    Joined:
    Jul 28, 2004
    Messages:
    246
    Likes Received:
    0
    I think it would be more interesting to not have any requirements like (only Windows, must use DirectX, only use 1 single 256x256 texture, out-of-the-box functions, etc.), but just say what the program must do (display 500 moving sprites, etc.) and let people implement this in the best possible way (as long as you provide the source code). It might be interesting to see how OpenGL compares with DirectX, and Windows vs. Mac / Linux.

    What exactly are you planning to measure? The number of FPS for 500 sprites? I think it would be more interesting to measure max. number of sprites possible in 100 FPS for example. Then there should't be a limit of 2000 sprites.

    I'ld split this into:
    - test 1: transparent sprites, no alpha blending
    - test 2: translucent sprites with random alpha (if supported)
    so you can compare this with DirectDraw / GDI / older 2D libs as well. When choosing a system/library for a game, you'll probably have decided wether or not you're planning to use alpha blending.

    Some libraries might have very slow text functions (since you'ld normally use bitmap fonts instead of text), so I'ld suggest to only display the last FPS value for a moment when you press F (counted without any text on the screen), or use bitmap fonts and display it real-time.
     
  7. dxgame

    dxgame
    Expand Collapse
    Original Member

    Joined:
    Mar 11, 2005
    Messages:
    267
    Likes Received:
    0
    hmm.. the more complicated it gets, the more likely no one will ever code something up. I still think a simple sprite render test using 32x32 sprites with random alpha sprites moving around via a pong style would be good enough. I also think any method is relevant, so if you have your own mod to an engine or whatever, let's see that as well. Source to me is not manditory. I'm sure if someone writes a demo that is performing ALOT faster than everyone else's inquiries will have to be made. ;)

    I guess allowing more then 2000 sprites is fine, but probably would only matter on higher end computers.
     
  8. Pyabo

    Pyabo
    Expand Collapse
    Original Member

    Joined:
    Jul 27, 2004
    Messages:
    1,316
    Likes Received:
    0
    I think this is a great idea...

    Coincidentally enough, I've already written most of this test to run under PTK, when I was evaluating it. I think the sprite I used is only 16x16, but they bounce around the screen pong-style... and I can adjust the number of sprites and change between fullscreen and windowed on the fly.

    Dxgame, since this is your idea... maybe you should write a template C++ program to run the test. Just leave some nice comment blocks where any library-specific code needs to go.
     
  9. Indiepath

    Indiepath
    Expand Collapse
    New Member

    Joined:
    Aug 22, 2004
    Messages:
    999
    Likes Received:
    0
    You might also want to test with a couple'a hundred thousand sprites since GPU tech nowadays is going to cope with pretty much anything you chuck at it.
     
  10. dxgame

    dxgame
    Expand Collapse
    Original Member

    Joined:
    Mar 11, 2005
    Messages:
    267
    Likes Received:
    0
    couple'a hundred thousand sprites? jeesh.. I pretty much run only on older gpu/cpu systems with onboard graphics capabilities. ;)

    "Dxgame, since this is your idea... maybe you should write a template C++ program..."

    I'm a VB6 person, sorry. I am going to throw together a demo to get things rolling. I'm sure my engine will probably be the slowest, because it's so high level, but oh well. Should have something ready later today.
     
  11. electronicStar

    electronicStar
    Expand Collapse
    Original Member

    Joined:
    Feb 28, 2005
    Messages:
    2,069
    Likes Received:
    0
    I have written a prototype of 2D engine that fills exactly these conditions (complete with the pong-like collisions) in JAVA (using JOGL for OpenGL).
    I'll see if I have enought time to clean it up and upload it.
    The 2000 sprites barrier should be no problem I seem to remember testing with up to 3000 alphaed sprites on an AMD 2600+ with GF ti4200.
     
  12. PaulModz

    PaulModz
    Expand Collapse
    Original Member

    Joined:
    Apr 7, 2006
    Messages:
    53
    Likes Received:
    0
    This is a great idea, but here's the rub; the harder you try to compare oranges to oranges, the further the stats drift from real-world performance.

    I think the most doable way to approach it would be to start with a reference implementation that everyone can copy and a set of assets that everyone can share. I'm currently moving my GDI+ vector graphics code to 2.5D using Direct3D, and I'd be happy to write a managed (C#/.NET) version.

    Don't use random positions, images or alpha values. The position, texture, alpha, etc. of each sprite should be identical each time the program is run, like 3DMark, with no user user input during the benchmark. You could put all the relevant sprite info into an XML or comma-delimted file to be sure everyone is using the same values.

    It would also be helpful to have the benchmark run a series of tests in multiple resolutions and windowing modes.
     
  13. electronicStar

    electronicStar
    Expand Collapse
    Original Member

    Joined:
    Feb 28, 2005
    Messages:
    2,069
    Likes Received:
    0
    I think the original idea was just to see different implementations in action, what would be the point to just copy and compile the same thing for everybody?
     
  14. Mike Wiering

    Mike Wiering
    Expand Collapse
    Original Member

    Joined:
    Jul 28, 2004
    Messages:
    246
    Likes Received:
    0
    Or use a very very simple random generator. That's alot easier than reading values from a file. Something like:
    Code:
    Global Seed = 0
    Function Random (X)
      Seed = (101 * Seed + (Seed Shr 5) + 5659) And $FFFFFF
      Return Seed Mod X
    End Function
     
  15. dxgame

    dxgame
    Expand Collapse
    Original Member

    Joined:
    Mar 11, 2005
    Messages:
    267
    Likes Received:
    0
    Looks like it's hard for developers to agree on the simple things sometimes. ;) In any event, here's a little demo I wipped up that shows what I was talking about:

    http://www.dxgame.com/download/SpriteTest.zip (400k)
    Requires Windows XP, DirectX8, VB6 runtimes (already on XP systems)

    F: Toggle Framerate Display
    S: Toggle Sprite count display
    Right Arrow: Add 100 sprites (up to 2000)
    Left Arrow: Remove 100 sprites
    R: Init sprites with rotation
    ESC: Exit

    Each time the sprites are initialized, a random sprite image is defined from the texture, a random alpha value, a random direction to Pong around the screen with.

    I threw the rotation option in just to emphasize a real usage scenario because I'm always rotating sprites in my games. ;)

    I already know my engine is slower than others, but I'm mainly interested to see how TGB, PTK, Blitz, etc, perform in a similar test.
     
  16. undersan

    undersan
    Expand Collapse
    Original Member

    Joined:
    Oct 25, 2004
    Messages:
    52
    Likes Received:
    0
    http://www.ericology.com/temp/SpriteMark_1_0.zip (537k)

    This is DX8.1 and C++. I wanted to see if I could reach my Radeon 9600 Pro's advertised peak fill-rate of 1.6 gigapixels/sec, so I went kinda overboard with the render-state options. I got up to 1.3, meh, pretty close.

    Note to benchmarkers: Even with the full display turned off, my app's framerate is still displayed top-right (only every 6th frame, to save time!)

    I haven't compatibility-tested my D3D framework, so I apologize in advance if this crashes on certain cards.

    As dxgame predicted, his VB6 version is slower than my C++ version. Here is the comparison on my system (2000 32x32 sprites, windowed):
    dxgame: (deleted, see my later post for correct results)
    undersan: (deleted, see my later post for correct results)
    Update: My initial testing was slightly flawed. I had the DirextX debug runtime libraries enabled--d'oh!
     
    #16 undersan, Jun 24, 2006
    Last edited: Jun 24, 2006
  17. dxgame

    dxgame
    Expand Collapse
    Original Member

    Joined:
    Mar 11, 2005
    Messages:
    267
    Likes Received:
    0
    Hi Undersan! Thanks for coding your example.

    "As dxgame predicted, his VB6 version is slower than my C++ version. Here is the comparison on my system (2000 32x32 sprites, windowed):
    dxgame: 56 fps
    undersan: 190 fps"

    hmm... this is a pretty big difference so I ran some tests on my system(s) to compare yours, especially ingteresting since they are both using DX8! ;)

    My results were radically different than yours:

    On my Intel Extreme Graphics Card my VB6 engine actually performed better than your C++ version, in both windowed and full screen with 2000 sprites.

    dxgame: 40fps
    undersan: 39fps

    I then tested it on my Raiden Express 200 Series computer, results were slightly faster but both engines were VERY close again:
    undersan: 69fps
    dxgame: 65fps

    THIS is the reason these kind of tests are great. For some reason my engine is performing poorly on your Radeon 9600 Pro compared to your engine. Perhaps there is a hardware render state I should be setting when running on a 9600 pro?

    Thanks again, lets hope some others chime in. This could be very interesting!
     
  18. Mike Wiering

    Mike Wiering
    Expand Collapse
    Original Member

    Joined:
    Jul 28, 2004
    Messages:
    246
    Likes Received:
    0
    Still, I think displaying the text influences the results too much in dxgame's program: with 100 sprites, I get about 320 fps if the "100" is displayed and 455 if not, but that is with fps display on. The moment I press F, it shows 550 or more for a second.
     
  19. dxgame

    dxgame
    Expand Collapse
    Original Member

    Joined:
    Mar 11, 2005
    Messages:
    267
    Likes Received:
    0
    "Still, I think displaying the text influences the results too much.."

    If I was using bitmapped fonts the performance hit would have been alot less, just a few more sprites then. ;) In this example I'm using the standard DX8 Text Rendering which is extremely slow and should never be used for anything serious in my opinion. I suggest using a 3rd party fps displayer like Fraps to help verify frame rates.
     
  20. Indiepath

    Indiepath
    Expand Collapse
    New Member

    Joined:
    Aug 22, 2004
    Messages:
    999
    Likes Received:
    0
    Here's the BlitzMax (DX7) Version : http://www.indiepath.com/tim/SpriteTest.zip

    T: Toggle FullScreen/Window.
    F: Toggle Framerate Display
    S: Toggle Sprite count display
    Right Arrow: Add 100 sprites (the sky is the limit)
    Left Arrow: Remove 100 sprites
    R: Toggle Rotation
    ESC: Exit

    #################

    Here are my results for 2000 Sprites Windowed

    - BlitzMax : 196FPS
    - DX Game : 94FPS
    - Undersan :108FPS

    Here are my results for 2000 Sprites FullScreen

    - BlitzMax : 230FPS
    - DX Game : 102FPS
    - Undersan : 119FPS

    FYI, I can run 10,000 Sprites in BMAX at 60FPS.
     
    #20 Indiepath, Jun 24, 2006
    Last edited: Jun 24, 2006

Share This Page