De-flickering and Beautifying Imperfections

I’m working on a deflickering / beautifying post-process compositing setup for animations, and would like your feedback. The denoising system is still in its initial stages. I’ll post its setup and blend file once it is at an acceptable stage. Both scenes were composited with the same node setup.



How can I improve the filters?

Also, let me know what you think of my de-noising process so far. The basic concept is as follows:
The initial rendering outputs a sequence of multilayer OpenEXR images containing the shading, normal, z-depth, mist, and vector passes.
In post-process, I use the vector pass to compensate for movement in the frame, and dynamically average the current, next, and previous frames. Finally, I find the closest consecutive pixel in the current frame to the averaged frame.
By adding the difference between this temporally filtered frame to the areas occluded in both the previous and next frame, I now have a mix parameter for bilateral filtering. (I would probably use some dilate / erode stuff on the bilateral masking factor).
Repeat for all shading passes (direct and indirect are separately filtered), and use un-processed material color data for everything.

Known weaknesses: Reflections and refractions need a very low color threshold to maintain sharpness during bilateral filtering. As a result, they are left more or less untouched by the filter as long as there is motion.

Both images were rendered with 256 samples (a more reasonable number for animations)