Can you have a shadow catcher that does not catch its own shadow?

Can you have a shadow catcher that does not catch its own shadow?

What I mean is can you have a shadow catcher that both casts and catches shadows but does not cast shadows on to it self. The reason being is that I am doing a match move animation with the new tracker and I am recreating the lighting with a reconstructed modle of the room. This because the light source is coming from a window and I want to get an accurate clipping of the light coming in through it. The problem comes when the mesh of the room is casting shadows on itself which in compositing just darkens the walls. I am looking for a way to accomplish this effect wether its through compositing or through conventional settings.

With ray shadows for the light off (there is some ao) the walls are ok

With ray shadows on, the back wall is shaded because there are drop shadows being casted onto the mesh that I have reconstructed

http://s16.postimage.org/y0d61ofc1/To_many_shadows.jpg