Shadow catcher and Ambient Occlusion issue

Okay so here is my problem, When using the shadow catcher I get additional shadows that I assume are ambient occlusion or the actual shading of the object but how do I turn this off and keep the shadows ? See image below.

There is only one light in the scene, the scene is a bunch of spheres on some steps

I’m encountering the exact same issue in my project right now. Did you ever find a solution to this?