Problem with shadow-only-plane and spotlight

Doing the trick where I put an object in real space, using shadow only plane to give real looking shadows.

I position the spotlight to the side where sun might be, with shadow only turned off, the shadow are nice and hard and on the side of the picture where they should be. When I turn shadow only on, the shadows rendered are almost non existent and very soft. Am I doing something wrong?