Shadow problems when adding an object to an background picture behind a hole

I have a background picture in which I want to add an object with blender.

The object should appear as if it is behind the hole.
I added a floor as shadow catcher which works fine.

The problem is that I can’t get it behind the wall. My first attempt was to add the bottom part of the wall as an object in Blender and select “shadow catcher” and “holdout”. This creates an alpha transparent hole in the render and you can see the background image as I want it. The problem is that the wall part creates a shadow on the floor from the wall, which I don’t want. I just want the shadow from the monkey.

I tried deselecting “shadow” in “ray visibility” for the wall, but it didn’t work.

Any ideas how I can solve my problem?

Wall shadow I don’t want