Actually by self-masking I mean that the masking needed for compositing is made part of the rendering rather than being generated by some other process. Think about it – a head of hair cannot be simply rendered by itself with alpha and then composited, because you have to mask out the part of the hair that’s obscured by the head (let’s say the back). But this same hair will also overlap the head in front, such as bangs, and external masking for the head will obscure this as well. Having both obscured and overlapping parts means you can’t use simple alpha-masking.
Instead, imagine the head rendered in place with the hair. Then, no matter the view, the head will obscure the proper parts of the hair, and the hair will overlap the proper parts of the head. So now the problem is how do you remove the head portion and leave just the “self-masked” hair? One way is to use some form of keying like chroma key. If the BG and the head are exactly the same color, and this color is keyed out during the compositing, what is left is the hair, already masked by the now keyed-out head, ready to be composited on top of the head that was rendered separately using another process (in this case, Cycles).
The choice of the matte process used – chroma, luminance, or whatever – would be somewhat dependent on the specific nature of the components used for the composite, but the self-masking technique could be used in most of them.
The self-masking method has been around for a long time, from the days when the first computer-controlled cameras allowed making identical passes on model shots – Douglass Trumbull used it on many Blade Runner shots – and just needs to be adapted to all-digital workflow.
The critical factor in any matting/compositing process is a perfect match between the composited elements, so you’d have to make sure that all the camera settings in both the Cycles and the yafaray renderings result in a perfect match between the separately-rendered parts, in terms of field of view, perspective, depth of field, etc. This may present more difficulties than the actual matte/composite process.