I’m not sure if this is what we’re dealing with here, but I do know that you have to be very careful when dealing with objects that are rendered with their own lighting-artifacts: shiny edges, casting shadows, or what have you. Depending on how you algorithmically combine the layers, and how you mask them, you can “add-up light” in places such that it all looks wrong and it just doesn’t work. For example, I’ve made mistakes where two objects are passing one in front of the other, and highlights from the rear object can be seen as though they were passing over, or through, the front object. The alpha-information was being combined incorrectly.
You might find that you need to composite neutrally-lit objects, then add the highlighting and the shadows as a separate “beauty pass.” You can control the types of information that contribute to a masking channel, and you might need to omit things like specularity from that.