BI: How to tweak AO without aliasing appearing?

Support Compositing and Post Processing
#1

Hello everybody,

I’m fairly new to compositing in Blender but I think I know the basis. There’s a problem I’m unable to overcome at the moment.

Basically what I want to do is to to take the Ambient Occlusion render layer with Blender Internal, tweak it to make it essentially two-tone, then remixing it back for the final composition. This is needed in another work of mine to help achieving a different “toon” effect.

Here’s my current test setup. Please note that for more clean results I switched to the Gather rendering method. Ambient Occlusion is set to 1.0, Multiply in render options.

aotweak.png1858×1062 300 KB

In the rendered backdrop essentially it’s happening what I intended to do, with just one difference: aliased eges appear on multiple spots on object edges wherever there is AO. The blacker or less tone detailed I make the AO layer, the more aliasing appears.

Surely, this is probably the fault of my current note setup. It’s very simple, but that’s what I’ve been able to figure out by myself to make it work.

So, the question is: is anybody able to figure out a better set up to tweak the AO layer only without introducing ugly aliasing artifacts in the image and preserving the original alpha channel information?

EDIT: added .blend file
aotest.zip (85.7 KB)

#2

I’m not sure of the effect you’re looking for, but try turning on Full Sample antialiasing in the Render panels. You could also increase the samples in the AO Gather, but with your color ramp you’d probably lose the enhanced AO effect you show in your preview.

#3

Perfect, this solved the problem I was having :slight_smile:
…although with the viewer node it’s still visible, but that’s not important.

Thanks!

#4

What it looks like to me is that you are using the Color Ramp to compress the original AO tonal range, so all the fine gradations at object edges (from normal AA) are being compressed into only a couple or three tones of very high contrast, which won’t support a fully aliased look.

Multiply will make the problem worse, btw, because your very dark AO shadows are making the existing pixels at object edges even darker.

The only solution comes to mind if you need those very dark AO shadows is to do some masking that trims off the perimeters of the jaggy areas and leaves the original AO tonal range more or less intact.

Another possibility is to use Render Layers to separate portions of the rendered image and overlay them on the amplified AO image, maybe re-establishing the edge pixel values closer to the originals.

#5

Yes, compressing the AO tonal range is exactly what I wanted to accomplish. The side-effects, those I don’t want, though. I was expecting some aliasing on the AO contours, but not on edges where apparently there isn’t.

Multiply will make the problem worse, btw, because your very dark AO shadows are making the existing pixels at object edges even darker.

The only solution comes to mind if you need those very dark AO shadows is to do some masking that trims off the perimeters of the jaggy areas and leaves the original AO tonal range more or less intact.

Another possibility is to use Render Layers to separate portions of the rendered image and overlay them on the amplified AO image, maybe re-establishing the edge pixel values closer to the originals.
I think at the moment this goes outside my expertise, unfortunately.
Using the full sample option to antialiase all render layers (while using slightly less extreme tonal range compression and a less dark AO instead of pure black - which wasn’t my intention to use where I need this trick to appear) appears to be a simpler option.

#6

It’s possible to use some simple filtering to regain that solid black but keep it away from the aliased edges. Pipe your new compressed range into a Brightness/contrast node and adjust so that the edges are burned out but the centers are kept dark. The result may be a bit hard-edged so you may want to then run it through a Blur node to soften it a tad. Then Multiply this edge-trimmed plate with the not-quite-as-black-as-you’d like AO plate, and it should give you a very dense black, but not affect the now properly-AA’d edges. As a final step before compositing back over the main scene, you may want to use some Math nodes or maybe a Normalize node to limit the numerical range in the shadow image from zero (full black) to 1.0 (full white). All the other nodes can sometimes push the numeric values out of the “normalized” range, and that can affect the final composite.

Oh, also, the reason you see the jaggies where there apparently is no AO is because you’re not dealing with a 3D image anymore, but 2D, so the boundaries between picture areas become important even if the depicted objects are well-separated in the 3D scene. It’s much like applying filters in Gimp or Photoshop in that regard.

#7

^^^That’s a nice tip chipmasque, thanks for sharing it.