I suggest doing things like this as follows: use at least three layers. The first is the base set. The second is the object. The third is the shadows that are cast upon various things by the objects. If there are significant environmental shadows being cast onto the object, then use a fourth layer.
In the base set layer, there are no shadows (other than, perhaps, the shadows that fall upon the set). Things that will “cast shadows onto the set” are not here, either. In fact, this render might be “shadowless.”
In the object layer, only the object is there. Likewise, “shadowless?”
Shadow-layers are only for shadows, and their purpose is to describe the location and the opacity of the shadows. Although you will “see” the shadow-casting objects in this layer, only the shadow data is actually used. (Minimize the amount of calculation, e.g. by using shadow-only settings for this layer, because the only data you’re going to keep from this is, “shadows” and “alpha.”)
(In a “final” workflow, each of these render-layers would be individually sent to MultiLayer OpenEXR intermediate files by a number of distinct render-jobs. But, I digress.)
In the compositing noodle that brings all these things together, the shadow information will of course be used to reduce the illumination in an area, but it also might be used to regulate hue/saturation and contrast within those areas. You provide “knobs” that allow you to precisely regulate this effect for maximum impact. (And, generally, you define these as reusable node-groups, stored in libraries, so that you can be consistent from shot to shot.)
As I said earlier about “fourth layer,” keep each source … each reason for … “shadows” clean, pristine and separate. This allows you to blend them together without “doubling shadows.”
In most renders, “shadows are where all that time goes.” Shadowed areas are also often what prompts your eye to reject a shot. Well, now you can fix, make it more pleasing, without re-rendering. “Priceless.™”