Question on refining luminance key node result

In this scene I replaced the sky using the luminance key node, but along the contours of the foreground there is an ugly outline that wasn’t removed. What is this unwanted effect called, and how can I eliminate it?

This is a snapshot of the original video

Here is the video:


To provide better help it would be easier to see a screen shot of your node setup. But the key needs to be softened somewhat.

Remember that you are adding a dark background to a formerly bright part of the image. So what ever effect was there before (light wrapping around branches or semi translucent leaves, and in your case lens flare in the center of frame) will have to be modified.

Two things to try first are:

  • expand your alpha a bit and use it to grade the leaves so that they blend better with BG;
  • use addmix to control the edge. In blender you can produce addmix by altering the alpha AFTER premultiplication with a power function or curves node. SeparateRGBA the alpha out after premult, then alter it and then CombineRGBA again. Addmult changes the alpha values between 0.0 and 1.0 so that the blend will be darker or lighter. In your case you want it to be darker.

If it is unclear, what I mean by addmix, I can throw together an example.

Here is a snapshot of my node setup.

Yes, looks like expected. And the follow-up question is what exactly? Have you tried grading the leaves or other solutions?

The dilate/erode node, when it is set on “step mode”, has the right idea I’m looking for, but the effect is too strong. It looks like setting it somewhere between step 0, and step 1, would be a big improvement.

I would love to understand this addmix setup, please show me how it’s done.

I don’t think that altering the alpha will help as the problem is with the fill channels. They are to bright. You might need a mask for the flare around the leaves and grade that back before mixing it with the original plate (using original key of course). Sadly that will alter the aliasing around the fine edges of the leaves as well, so you may have to blur the result.

An example of basic addmix setup and what it does.

Addmix alters the alpha value of FG separately from the (inverted) alpha that is applied to BG. The logic becomes clearer if we look at the basic compositing operation (Over):

C = A * alpha + B * (1 - alpha)

Background is multiplied with inverted alpha and added to foreground multiplied by alpha. This is the most basic compositing operation used when dealing with transparency. As it is, all resulting pixel values where alpha is between 0.0 and 1.0 will be a weighted blend between FG and BG. This is all nice if the original FG element does not have contamination from different background.

The problem with compositing over rises when we have original BG luminance and color baked into FG pixels. The alpha can be 100% accurate, but the A element itself is not, and so the result will be either too bright, dark or otherwise wrong. This is a problem that can be deceivingly easily solved by eroding the alpha in. But eroding ruins your edges and can introduce nasty artifacts.

Addmix comes to rescue. Addmix alters the alpha for A and B multiplication separately, so that A or B side can get more or less weight and thus the result will be darker or lighter. The only thing that must be done is altering the alpha in a way that alpha and its inverse do not add up to 1.0 anymore in semitransparent areas. One can use Curves, power function (gamma) etc, whatever seems fit. But 0.0 and 1.0 values must remain fixed, so multiplication and addition do not work.

An explicit nodegraph for addmix in blender:

And the results on some arbitrary ugly image:

And a bit different setup that does the same thing:

The addmix was helping, but I found a way to remove all the ugly outlines, and at the same time lower the brightness of the trees. Now the foreground is much better integrated. I was thinking that I could fix the brightness of the trees by using a color mix node set on multiply, and in one input, a blend from very dark grey, to white, from the top to the bottom of the image. I made the setup, the brightness looked much better but I was surprised when all the outlines were also removed.

Nice work… now to be picky, the trees in the mid ground have darkened as well, but they should have more light on them. For extra points try and work out how to mask them from your gradient effect (hint: subtract)