Hi,
Over the weekend I was playing with the notion of using motion vector controlled temporal filtering in order to get rid of high frequency noise in image sequences and since I’m happy to say that the experiment was a success I’d like to share my findings with the community.
The idea is basically this: instead of waiting a certain amount of time to get a clean render of an image sequence, we can render the image sequence in much less time (between 2 to 4 times for a path tracer like Cycles) and then using temporal filtering controlled by motion vectors (which are free in Blender as well as in a lot of other renderers) reduce the noise to a (hopefully much) lower level. For a given frame the denoising is done by transforming the previous frame and the next frame into the current frame using their motion vectors and then averaging the three frames together (practically doing a temporal box blur). This process can be repeated several times on the already filtered result in order to get a wider and softer filtering.
Here are some example images (the render is a single frame from an animated version of the default Cycles scene) - the first is a Cycles render using 10 passes, the second is a Cycles render using 30 passes and the third is a denoised version of the 10 passes render:
This approach, beside denoising, could also be used as for firefly removal by applying a temporal median filter instead of a temporal box filter on the three images (unfortunately this is not currently supported by Blender).
I borrowed the idea behind this technique from The Foundry’s Furnace DeNoiser plugin which denoises film or video recorded sequences by doing an optical flow analysis on them and then uses the resulting motion vectors to drive a temporal filter.
Since this is a post technique it’s entirely done in the Blender compositor, but it needs the 3D scene in order to get the motion vectors from the BI renderer. I hacked the Vector Blur node, so that instead of doing a vector-based blur it does a vector-based image transform operation. The biggest problem (not counting my lack of familiarity with Blender’s code and lack of C/C++ programming knowledge) was dealing with occlusion artefacts, for which I didn’t use the most elegant approach, but as far as I can tell it works (any transformed pixels with sides that are longer then a certain number of pixels in the resulting image are discarded).
I haven’t done a lot of testing and there are noticeable artefacts in the example image (for ex. the edges of the plane in the lower half of the image), but these are implementation issues which I’m sure can be sorted out.
Since the whole technique is practically a hack, there are quite a few limitations, the biggest being that when a 3D object’s surface changes appearance drastically between adjacent frames (for ex. animated textures which have big changes from one frame to the next, etc.) the technique would produce visibly incorrect results. Some of the limitations can be overcome simply by adding motion vector blur to the denoised images or by rendering the image in passes and then applying the denoising process on the indirect diffuse and the glossy reflection/refraction passes which most of the time would be the noisiest and at the same time would have the least high frequency content.
Unfortunately, I’m pretty sure that I won’t have the time to continue implementing this approach into Blender, so if anyone is interested to continue along this way feel free to send me a PM, so I could pass you the torch
Thanks and sorry for the long post,
Goran