It will be great when Blender gets “light linking or including/excluding” but Blender does still have “scene specific lights.”
EDIT: I have corrected these are “SCENE” specific lights not “LAYER” specific.
Each scene can have its own lighting setup. With a shadow catcher and a bit of a workaround you can do something like what you are after.
Here I have 3 scenes composited together.
One with a ground plane, a cube, a cylinder and 2 lights (whithout shadows)
One with a shadow catcher instead of the ground plane, the cube and one light (with shadows)
The last with a shadow catcher instead of the ground plane, the cylinder and one light (with shadows)
I included the cube and cylinder to the first scene (with the ground plane) to be able to fade the shadows a bit with the factor of the alpha over node.
In this version I changed the lights from sun to spot and gave them different colours so you can see how each light contributes to each object.