I am trying to add a glow effect to an object, but have the problem that this object is being viewed through a glass material. Thus it seems impossible to correctly produce an idmask.
If the glow object and glass object are in the same render layer, then I only get the bits not behind the glass.
If I put the glow object into a separate render layer then the idmask corresponds to a shape that does not take into account the refraction produced by the glass.
The only way you’re gonna get this is by keying it out. Why not try using a pre-rendered image-plane of the object with the glow already on it attached to an image which is then placed behind the glass. If you have a moving camera then you’re image is gonna have to be a movie on the plane and the plane will have to be constrained to always face the camera.
There’s probably not a realistic way to code that since the object pass is based on the z-buffer from camera view and the object that you’re wanting the pass for is occluded by the z-buffer of the glass in front of it. It’s akin to the noobs on this forum always wanting alpha transparency and z-buffers to work together but by definition these two are mutually exclusive passes and cannot work together. What you have to remember is that this is POST-PROCESSING and as such layers/passes are not combined in any way similar to the internal render process because no actual geometry exists in the output image. Post is all numeric 2-D data no mater how you slice it.
Like all other layers, an ID-mask layer will only show you information for things that the camera “can see.” Therefore, just create another RenderLayer … one that doesn’t “see” the object that’s in-the-way. (“To this layer, the glass doesn’t exist.”) This will produce an ID-mask matrix that you can use.
… yes, I see that (now) … and there really is nothing that can be done about it. You’re going to need something that represents, two-dimensionally, the net effect of the refraction as viewed from the camera image-plane. Perhaps a plain-white stand-in object could be used. Perhaps you could use chroma-key. But an id-mask is simply going to show you what pixels are visible from the camera image-plane as though nothing else were in the way. If that “thing that is in the way” modifies the path of the light, and hence the fundamental shape of the object as viewed by the camera, then you’re going to have to do that math in order to come up with some kind of mask that can be used for compositing.