Track an object in a video?

Does anyone know the best way to track an object in a video?

Let me try and explain:
I have a video set as a world texture, and I need to track the motion of someones hand in the video. The idea is that they catch a CG object and hold it. So far I’ve tried positioning an Empty on the hand, inserted a key frame, and then advanced a few frames, inserted a key frame, etc. But when I play it back the Empty floats around the hand too much.

Is there a plugin or 3rd party software that tracks objects in videos which can then be imported into blender? Or do I have to do it the hard way and keyframe every frame?

I’ve had a look at Voodoo, but that only tracks camera movement.

Icarus and Voodoo are 3rd party software you can use for motion tracking.
Basically icarus allows you to track the motion of the camera and exporting that camera movement to Blender.
Also check here for CG compositing with Blender.

If the object is moving to much, then try changing the interpolation mode in the ipo controls from bezier to linear. I have found this trick to make the tracking more accurate when keyframing close together.