I am trying to use composite nodes to combine different render layers. As I posted last week, edge render, renders right over the top of hair particles so I need to render them in different layers and combine the renders - the best way to combine them seems to be Zcombine but I’m having some problems; here is my experiment:
I have 4 layers, first is a blue cube, 2nd is a red sphere centered at the same spot as the cube, 3rd is a backdrop curved plane, 4th is the camera, a white area light and a pink spotlight.
when I click on all the layers and render this is what I get (and this is what I am trying to recreate using the seperate render layers).
As shown in the render of the backdrop, even though the cube and sphere aren’t in this render layer, they still cast shadows here.
So when you just render the cube, you see the shadow of the overlapping sphere
and likewise with the cube shading the sphere
here is my composite node setup
and when you do Zcombine of the sphere and cube render layers you get what you’d expect but you can see a dark line where the cube and sphere meet(this is the output of the first ZCombine, top second column of nodes)
If you combine the alpha layers of the sphere and cube render layers you get what you’d expect
So now my goal is to combine all three render layers to make the output look like the render when everything was rendered simultaneously (the first image I showed). I could think of three ways to do this.
The first way is to simply use another ZCombine node, and the colors look good but there is a black outline around the edge of the sphere and cube when I do this (this is the output of the node highlighted in the node diagram)
So in an attempt to improve this I thought that using the alpha layer would be ideal. If you simply mix the backdrop render with the ZCombined layers using the added alpha layers as the factor then the dark line around the outer edge of the ZCombined objects disappears but the inner dark line remains and the color is too intense and it looks like the color from the cube is added into the image as well as a portion of the shadow from the backdrop plain.
A similar result comes when you use alpha over. Note that ConvertPremultiply is clicked to prevent alpha from being greater than 1.0 and that was only noticeable in one small porion of overlap so that’s not what is causing the color / saturation error.
So in summary, I can use ZCombine nodes to combine several render layers but I get an artifact around the edges of each object rendered seperately. Any ideas how to prevent this edge artifact?
This is because the Z pass is not anti-aliased. Z just picks the Z value of the most occluding face in that pixel – there is no interpolation. Not sure why you’re trying to use Z-Combine to do this. If I were doing it, I’d break the pieces out on different Render Layers like you did…
- In the Render Layer controls for the box, Ctrl-LMB click on the (unselected) layer button for the layer that contains the cube. This should put a black spot in the button.
- Combine the two passes (Renderlayer-Box and Renderlayer-Sphere) using an Alpha Over node.
- You will notice that the pass with the box now has an alpha knockout for any objects in the “black spot” layer that occlude it from the camera’s perspective.
- To get a proper combination, use either “Premul” or “Key” sky mode
In short, Z-Combine isn’t really intended for slicing a scene and putting it back together. It’s useful for other things, but AlphaOver and the Ctrl-LMB “layer masking” technique is the preferred method.
turn off OSA. To do your own osa on the composite, render at 2x and scale the result down.
aahhhh thank you. I am just getting into using render layers. Could you recommend a tutorial discussing layer masking?
Any idea why my alpha over seemed to go so strange? It looks like a partially masked color layer is added and the shadows are subtracted!
well, <promo>my book Foundation Blender Compositing has chapters on masking and compositing. Pick up a copy at your local bookstore. </promo> 
Harkyman was the technical editor, so he will vouch for me.
LOL I just finished placing the order, good timing!
Of course, <competition>the technique I just described is also described in the compositing chapter of my own book, which is available in my sig.</competition> It deals more with larger, complex projects.
But I do vouch for PapaSmurf’s book too!
OK I suppose that I’d better just get them both, sigh