Random crashes?

Discussion in 'Game Development (Technical)' started by ggambett, Aug 14, 2006.

  1. ggambett

    Moderator Original Member Indie Author

    Joined:
    Jul 26, 2004
    Messages:
    1,982
    Likes Received:
    6
    As you may have read before, any time one of my games crashes, I get a crash report including a stack trace in my inbox.

    This has helped me immensely in debugging the games - almost every game logic bug is gone. Now most the reports I get are weird crashes, such as segfaults in relatively innocuous destructors (ie when the game is shutting down, a String destructor, or a Hash destructor that just deletes its nodes).

    Today I realized that most if not all of the weird crashes come from the Windows versions, even though I get a similar number of Windows and Mac downloads; and despite the fact that ~95% of the code is shared among the Windows, Mac and Linux versions (and the remaining 5% is usually not where the crashes are).

    So with the same code base I get weird crashes in Windows but not in Linux (which I use all the time while developing) or Mac. Valgrind doesn't detect any memory problems. I have a few far-fetched theories such as compiler bugs or OS memory allocator bugs, but obviously the sane thing to do is to avoid considering these possibilities, and work assuming the bugs are somewhere else.

    Does anyone have a theory, no matter how improbable it may sound, about what can be causing this crashes? Does any of you have any experience with similar situations?
     
  2. TheMysteriousStranger

    Original Member

    Joined:
    Jan 17, 2006
    Messages:
    222
    Likes Received:
    0
    Here's an out there theory:

    The bugs are the same on all platforms, it's just that the windows platform is better able to detect them?

    Just a thought.

    Also, are you using a lot of STL code? Like vector<T>? I have a weird one in my current project where after a lot of incremental compiles, the linker gets all confused and suddenly these containers start throwing exceptions, usually at either creation or deletion. However, a completely clean rebuild of everything usually fixes this.

    I also had an odd one where if I tried to fill a vector with more than 300(ish) objects without reserving the space, then that would cause crashes. Simply reserving enough space for all the objects fixed this.

    Have you any more specific/detailed crash examples that might help to narrow down the problem?
     
  3. ggambett

    Moderator Original Member Indie Author

    Joined:
    Jul 26, 2004
    Messages:
    1,982
    Likes Received:
    6
    It doesn't look that way. The fact that the games in the other platforms continue to work says, to me, that the bugs aren't happening, not that they aren't being caught.

    I have a feeling it's memory corruption... but the fact that I can't reproduce them on Linux (not a single time) and the fact that Valgrind doesn't detect any errors really confuse me.

    I'm not using any STL, by the way.
     
  4. Emmanuel

    Moderator Original Member

    Joined:
    Nov 23, 2004
    Messages:
    859
    Likes Received:
    0
    What compiler are you building the Windows executable with? I've experienced actual code generation bugs, as in the disassembled code at a weird crash site not matching the C++ code (or actually making sense), with VC++6.0 (upgraded to VS since then..) and with gcc 2.9.x / 3.0.x (then I lost track of gcc altogether).

    Best regards,
    Emmanuel
     
  5. ggambett

    Moderator Original Member Indie Author

    Joined:
    Jul 26, 2004
    Messages:
    1,982
    Likes Received:
    6
    Windows : cl.exe 13.10.3077 (the one that comes with the free command line tools)
    Linux : gcc 4.0.0 (RH 4.0.0-8)
    Mac : gcc, almost certainly 4.something
     
  6. sulaiman

    Original Member

    Joined:
    Aug 2, 2004
    Messages:
    20
    Likes Received:
    0
    It may have something to do with the external environment.

    Perhap some spyware/virus or anit-spyware/virus program that has corrupted the kernel and destablized the system. Or some seemingly innocuous program has a side effect that cause strange behavior.

    One example of an undesired side-effect is that when PSP7 is running, it doesn't allow my computer to hibernate for some reason. Hopefully this was fixed in the later versions.
     
  7. GBGames

    Indie Author

    Joined:
    Jul 27, 2004
    Messages:
    1,255
    Likes Received:
    0
    If Valgrind doesn't find it, then I don't see how your code would create the problem. Then again, your crash reports don't necessarily tell you anything about reproducing the errors, do they? Does it seem to happen without any kind of situation occurring, like a certain object getting created after another object?

    Usually frustrating crashes such as these almost always are a result of deleting something twice.

    At my day job, I just had a frustrating bug that stopped productivity for an entire day. I could recreate it, but I couldn't figure out what was going wrong. I commented all lines of code that involved deleting, then uncommented one line at a time, recompiling each time. The code was pretty much the same as before, but somehow the crash stopped occurring. I have no idea what was different to fix the problem. It makes me uneasy to think about it.

    The interesting thing? Valgrind wasn't a help. I couldn't get the crash to occur while using valgrind, and so I didn't know where the error was being introduced.

    Then again, development tools at my day job are not exactly up-to-date.

    The only other thing is: how many people are affected? Maybe they have a virus or something else running on the system, and your game crashing is a symptom.
     
  8. mahlzeit

    Original Member

    Joined:
    Sep 26, 2004
    Messages:
    852
    Likes Received:
    0
    Random crashes are often thread issues. Maybe worth looking into.
     
  9. lapskaus

    Original Member

    Joined:
    Jun 16, 2006
    Messages:
    51
    Likes Received:
    0
    It might be worth trying to compile the windows version with gcc too, since it's the version not compiled with gcc that has the crashes.
     
  10. ggambett

    Moderator Original Member Indie Author

    Joined:
    Jul 26, 2004
    Messages:
    1,982
    Likes Received:
    6
    No, not at all, and that's part of the problem... it's trivial code that works all the time, it works every time for me, but I still get crashes. That's why I'm pretty sure these aren't logic bugs. And there are some hints that make me think someone (my code, library code, or someone else) is overwriting my memory :(

    Probably... but I can't, for non-tech related reasons :(
     
  11. lapskaus

    Original Member

    Joined:
    Jun 16, 2006
    Messages:
    51
    Likes Received:
    0
    Oh well, it IS windows, and anything can crash on windows at no fault of it's own. My game started crashing randomly when i installed fraps. Even when fraps wasn't running. When it was running, the game wouldn't even start. Uninstalling it fixed the problem. I didn't check to see if it affected other games though.. i should do that.

    If your customers aren't complaining, i wouldn't give it any more attention. It's probably some problem with their system setup that they have to fix themselves.

    Non tech reasons? You got me even more curious there ;)
     
  12. MibUK

    Original Member

    Joined:
    Feb 8, 2005
    Messages:
    84
    Likes Received:
    0
    OK, this sounds obvious to me, so I cant think why nobody else has suggested it.

    A piece of your code that is inside an #ifdef WINDOWS block is writing to memory it shouldn't, either accessing a pointer that may have been deleted, or going over the end of the array.
    Valgrind under linux wont find it, because the linux version wont compile those sections in. (and as far as I remember valgrind wont run on windows under cygwin, maybe I'm wrong there).

    Use your own memory manager, and add sentinal bytes, produce a testing version that at every free verifies that the sentinal bytes are unwritten, and write a custom delete operator that sets deleted pointers to a known memory location that can be monitored for writing.
    Or find / buy a good windows memory check system, there are some out there, Rational purify comes to mind, but I think theres another couple of nice ones that dont cost quite so much.

    But thats my suggestion, it only happens in windows, but crashes happen in innocuous code, almost 99% of time I've found thats due to either writing to a deleted pointer or going past end of an array / string. Most of the time it wont cause an immediete crash, but will upset the memory manager so that a later free (destructor) or new will fail.
     
  13. zoombapup

    Moderator Original Member

    Joined:
    Nov 25, 2004
    Messages:
    2,890
    Likes Received:
    0
    get a trial of Boundschecker? I know they do a trial version for 30 days or some such.

    If you can, get purify instead, its better, but I'm not sure they do a trial.

    One other thing. Try stressing a test machine while running your game under windows. In particular, try using a memory stresser. It could be a failed allocation under windows (i.e. having more apps running using more memory up).

    Just remember, its not like you are the only thing on the machine. You really need to reproduce the environment the machine is running in. Send the customer who had the crash a build with machine info logging?
     
  14. Davaris

    Original Member

    Joined:
    Sep 29, 2004
    Messages:
    767
    Likes Received:
    0
    I had this same problem when I converted my game engine to use the PopCap framework earlier this year. I asked everyone and hunted high and low for the cause, because the stack trace files were not being created. Then I tried turning all of the settings off in Armadillo and the random crashes went away.
     
  15. Huge

    Original Member

    Joined:
    Sep 22, 2005
    Messages:
    142
    Likes Received:
    0
    Hi,
    Some ideas on differences you can find between OS/compilers:
    1. Allocation/deallocation order of static objects:
    Solution : no static objects - not great, but you can have static "simple" data - eg pointers. Use the (overused, IMHO) Singleton pattern to overcome.
    2. DLL unload order. This can be tricky if you pass pointers between DLLs. Also if you call system functions implicitly when an object destroys itsself from within a DLL as the DLL unloads itself. Really hard to debug too, because it can be hard to catch the DLL load/unload code.
    Here's an interesting though: not to do with you trace code itself ? ie, tracing some destuctor, after the trace code itsself has destructed.
    3. Window events coming in after you have destoyed an object. eg, close your window, destroy your main application and then a mysterious resize event comes in on the closed window and hits an invalid callback function.

    One possible way to prevent this, without perhaps even finding the exact problem, would be to explicitly clear all your vectors etc. on the way out, rather than letting them go implicitly at the OS's descretion. eg: global_list1.resize(0); global_list2.resize(0), ... etc.
     
  16. ggambett

    Moderator Original Member Indie Author

    Joined:
    Jul 26, 2004
    Messages:
    1,982
    Likes Received:
    6
    Interesting, I hadn't thought about static object destruction order. But even if different compilers do it in different order, shouldn't it be deterministic for one given compiler, and therefore reproducible?
     
  17. Moose2000

    Moose2000 New Member

    Joined:
    Jul 3, 2006
    Messages:
    138
    Likes Received:
    0
    If the destructor order is causing memory trampling (say, one destructor accessing a pointer left dangling by the destruction of another object), then you could have something which only outright crashes on a particular machine configuration, or when exiting from a particular state.
     
  18. jcottier

    jcottier New Member

    Joined:
    Jul 12, 2006
    Messages:
    1,385
    Likes Received:
    0
    You don't need to create your own memory managment on a PC software. The windows one is good enough. Also, it allow you to do what ever you want to track bugs.

    You need to check this:

    http://msdn.microsoft.com/library/default.asp?url=/library/en-us/vclib/html/_CRT__CrtSetDbgFlag.asp

    also have a look here:

    http://msdn.microsoft.com/library/d...s/vsdebug/html/_core_Using_the_Debug_Heap.asp

    Sometimes you have memory trashing but you don't know about it because the game doesn't crash on exit.

    JC
     
  19. Huge

    Original Member

    Joined:
    Sep 22, 2005
    Messages:
    142
    Likes Received:
    0
    The static object order should be the same, but the crash might occur only if some random memory was filled with 1s or 0s etc, which is possibly deterministic, but probably chaotic at best. Some things like a four letter machine name crashes then system eventually, but a five letter name is ok etc. Also uninitialised memory, or accessing dangling memory, can crash or not depending on the debug setting. Your release crashes, so you turn on debug and it works ! I hate that.
     

Share This Page

  • About Indie Gamer

    When the original Dexterity Forums closed in 2004, Indie Gamer was born and a diverse community has grown out of a passion for creating great games. Here you will find over 10 years of in-depth discussion on game design, the business of game development, and marketing/sales. Indie Gamer also provides a friendly place to meet up with other Developers, Artists, Composers and Writers.
  • Buy us a beer!

    Indie Gamer is delicately held together by a single poor bastard who thankfully gets help from various community volunteers. If you frequent this site or have found value in something you've learned here, help keep the site running by donating a few dollars (for beer of course)!

    Sure, I'll Buy You a Beer