Ok first off - if you're going to reply "use 3d", then please don't. 
Alright, I'm doing silly stuff - I'm trying to have a "3d" space simulation underneath the hood of my 2d isometric engine. This is so I can do stuff like bullet trajectory simulation, 3d picking, etc..
Now, I've got it working pretty decently in some ways, but I've hit the wall on something stupid. I've got an Isometric projection matrix, like this:
Code:
ISO Projection
0.70710 0.35350 0.00000 0.00000
0.00000 -0.86600 0.00000 0.00000
-0.70710 0.35350 0.00000 0.00000
0.00000 0.00000 0.00000 1.00000
That's dimetric projection actually, but you get it.
Now, that works just fine. I can render cubes around my ingame 2d objects in the proper world space. However, now I need to be able to unproject so that I can ray pick (target) with the cursor. I can get the coords in the traditional 2d way, but that won't respect collision hulls in the world sim and will give me incorrect results.
So, I get a transposed projection matrix (since I can't seem to invert it, and anyway it's just a rotation matrix so it should work fine transposed):
Code:
Transposed ISO Projection
0.70710 0.00000 -0.70710 0.00000
0.35350 -0.86600 0.35350 0.00000
0.00000 0.00000 0.00000 0.00000
0.00000 0.00000 0.00000 1.00000
But, it doesn't. I get mangled numbers out of that.. If I project the screen coords of 500,0,500 and then unproject them I don't get the original.
Anyone got a better handle on the math here? I can't quite figure out what I'm doing wrong.