I was having an issue recently where setting the substeps high (to get accurate simulations) caused the physics to become unstable. It wasn’t a complex system either. Turns out that if you have any triangle-mesh on triangle-mesh physics then you’ll have instability - just most of the time it’s too small to be noticable.
Both objects are triangle mesh physics type (they need to be in the actual scenario), and I’ve turned up the margins so it’s really obvious what is happening.
I don’t recall it being this unstable in the past, so maybe it’s just me (although I tested in three versions of blender). When you play the above blend, does the cube jump up and down vertically?
I noticed instability with triangle meshes as well, when I tried using them for a first person character (in order to make him crouch). The Character Object just started bouncing up and down vertically at a fast rate.
Gave up on triangle meshes after that.
I get similar instability in 2.49, though not near as jiggly. Word on the street is that dynamic triangle meshes aren’t supposed to collide with other triangle meshes, very un-optimized or some such. According to bullet docs they are specifically meant for static meshes. But you can build custom compound shapes using box or convex hull for better performance + stability.
Haidme linked this awesome video on BGE physics awhile back. It changed my life lol