I remember.Use a second empty for original location and ad the same constraints and use the same object as a driver just use negative number so when you turn on one drive you turn off the other so when it is released from one empty it goes to the other. the other empty is where the object will go. If you put the hand empty right on the object location when you move the drive there is no jump. that is why the object location is on the surface of the object. When you put the hand near the new location you can drop it with few tweaks or just place it on the location of the object (goggles) location.
A empty for the hand and one for the googles one switch to make it jump from one empty to the other empty.
I hope that is understandable.
Here is a quick job I did not have time to ad a second empty. When you release the object from the hand with the driver it will go to the head you need a empty with copy location and rotation for the head gear to go to, a new location.
Move the drive to the right. http://www.pasteall.org/blend/39252