wobbling around rigid body constraints

I’m trying to set up a simple animation: two bodies joined by a hinge constraint are thrown onto a plane. The constraint is specified for an empty that is a child of one of the bodies. Upon collision, the bodies wobble a little around the constraint point, like a wheel on suspension.
Is there a way to get rid of this wobbling?

Can you show us your blend file?