Yes, it’s an unfortunate consequence of the way holdouts work. Let’s step through the math in an example:
In this image, rendered all together, a pixel on this soft edge where Suzanne overlaps the sphere is 50% Suzanne, 50% sphere: half gray, half red, solid alpha. All good, as expected.
Now, let’s render with holdouts. In the sphere render, this pixel is now half sphere, half holdout: 50% red, 50% transparent.
In the Suzanne render, the same pixel is half Suzanne, half background (aka holdout): 50% gray, 50% transparent.
Now let’s look at what happens as we layer these up. We put the sphere over the background, and since this pixel is half-transparent, we get a 50-50 mix of the sphere color and the BG color: 50% cyan, 50% red.
Now we add Suzanne, and since this pixel is half-transparent, we get a 50-50 mix of Suzanne’s color and what’s behind her: 50% gray, 50% [mix of red and cyan].
One way to look at the problem is that the two elements are both holding each other out: even though the sphere isn’t actually eating away at Suzanne, the way we think of a holdout, there’s an alpha hole in the Suzanne layer in the shape of the sphere (and everywhere else too, but along that edge it’s the same difference), and there’s an alpha hole in the sphere layer in the shape of Suzanne. The semitransparency is being accounted for in two places, so it’s doubling up.
One thing you can do is add the two elements (and their alphas) together before layering them over the background: since they’ve both been reduced by the amount of the other, adding takes them back to full strength (minus imprecision from sampling noise: you can see a tiny bit of cyan peeking through in spots since the summed alpha only came to 0.97-whatever in those areas).
This works fine with solid objects, although I don’t think it’d work in your case, since it assumes both sides are holding each other out, and I assume your sphere render isn’t held out by the volume.
One thing that might work for you here is a disjoint-over: an over operation with a couple extra steps meant for double-holdout scenarios like this. The formula is
A+B(1-a)/b, A+B if a+b<1, where
A: the channels of the FG input
B: the channels of the BG input
a: the FG’s alpha
b: the BG’s alpha
which can be done in any program, since it’s just basic math operations.
In Blender, it would look like this if the Mix Color node operated on alpha,
but since it (frustratingly) doesn’t, it looks like this, which is just the same thing but manually per-channel. If anyone has a better way, I welcome it!
Regular over:
Disjoint-over:
