Ragged mesh intersections on rendering

[chipmasque]

This is a problem I’ve tried to solve or at least minimize for quite while but no luck:

The bra mesh intersects the figure mesh and things get really ragged where they meet. I’ve tried folding the edge of the bra mesh under so it doesn’t intersect but that produces some real difficulties when animating. I need to have the bra look like it has some thickness, 'cause even in long shots it looks wrong otherwise.

Any idea about how to clean up the rendering where the meshes intersect? Higher OSA softens the raggedness but doesn’t eliminate it.

[JenniferBlender]

Is ‘Full OSA’ selected on your materials ‘Render Pipeline’ panel?

[chipmasque]

@JenniferBlender

Is ‘Full OSA’ selected on your materials ‘Render Pipeline’ panel?

Actually, no it hasn’t been, since I was unclear on its benefits and it does seem to increase render times somewhat. But some test renders of the same frame with Full Osa enabled on both materials at the intersection shows that it doesn’t fix the problem – the ragged edges are softer (sort of like upping the OSA sampling value) but not eliminated. But thanks for the suggestion.

A half-day of running tests on Materials, Mesh and Lamp settings seems to show that this is not something that can be fixed with any one setting. I did discover that the the angle of the lighting has some effect – when the edges of cast shadows don’t fall very near the mesh intersection, the raggedness vanishes.

However, by adjusting the lighting so no shadow is cast, the ragged mesh intersection is clearly visible:

However, the angle of the mesh intersection may have some bearing on the problem, since the raggedness does not show badly at all places along the intersection of the meshes.

Here’s a theory on the cause, and why it looks worse when shadowed: At the intersection, the rendering engine has to make a choice whether a pixel being rendered belongs to one mesh or the other. Within a range spanning a number of pixels on either side of the intersection, the color and shading of the pixel to be rendered is ambiguous. So some pixels in this range get assigned to one mesh, and other pixels to the second mesh, but the choices seem to be a bit random within that range, leading to the raggedness. When shadows fall in this area the effect is magnified by both the contrast between lit & shadowed areas, and maybe also by a similar ambiguity, which may be of a larger scale, in the calculations of the shadow’s soft edge.

It doesn’t seem to be tied to the mesh resolution – when I upped the Subsurf level of the figure mesh, it made no difference at all in the appearance at the intersection. Changes to Camera settings (Lens, Clip, etc.) also had no effect.

In another 3D app I used in the past, there was a setting for the renderer that would create a small-scale gradient blend/blur between intersecting objects, smoothing over this kind of raggedness to a large degree. Is there anything like that in Blender’s toolset? Or, is there some way to detect mesh intersections in a way that could be used in the Compositor to modulate a Blur node to sort of smear the problem out a bit?

[rawpigeon]

Hi,

Are you using ray or buffered shadows? If you’re using buffered shadows, try ray shadows to see if they are the cause.

[MadMesh (Niels Philipsen)]

I’ve had similar problems but it was almost always related to Z-transparancy. Maybe enabling/disabling Ztra on the bra and skin material wil make a difference. Otherwise comping the bra with an IDmask could do the trick in the compositor.

[chipmasque]

Ray shadows from a large (Size = 100) Area Lamp, 8 samples, Constant Jittered (the other options here didn’t help), Umbra and Noise enabled (disabled – no diff; Dithered – no diff). Tried many different options, so I don’t think it’s the Lamp(s) but rather how they interact at the mesh intersection. Also, the prob shows when the intersection is not shadowed.

Toggled ZTrans on both materials also – nada. The IDmask sounds like something to try, since I haven’t already. Not sure yet how to do that but I’m sure I can find out.

[chipmasque]

Figured out the ID mask node – looks to be very useful, except…

The mask from the figure and that from the bra both show the raggedness!

This is contrary to what the manual says should happen – they use an example that does exactly what I’d like to do, but for some reason I’m not getting a smoothly aliased edge to the Alpha mask where the intersection is.

I also tried using the ID mask output as illustrated in the Manual and it doesn’t seem to work as shown – after trying a few different node combinations, I finally got a result that matched the original input in terms of image qualities, but the raggedy intersection edges were not modified much at all, still very evident.

[MadMesh (Niels Philipsen)]

I think youre right its not a lamp problem but more of a geometry problem. The only other thing I can think of is to apply the subsurf modifier and maybe even traingulate the mesh (no quads).

If that doesnt help you could try the developer forum, because this smells like a bug.

[chipmasque]

Can’t really apply the subsurf since this is being animated – the UI crawls with subsurf on, would make posing basically a nightmare. Triangulation (without applying the Subsurf) would create general chaos with the Subsurf, I think – the few small places I have tris now are always problematic.

I’m doing some more testing now, and it is very much linked to the geometry of both meshes – the most significantly ragged parts of the rendered intersection correspond exactly to where the mesh lines (the non-subsurfed lines) of one object meet the surface of the other (animated GIF):

The mesh lines seem to be acting like small ridges in the surface where the two meshes meet.

Not sure if this would be considered a bug or not – mesh intersections have always been tricky for rendering engines that do anti-aliasing, since they’re not really the same thing as meshes occluding one another. I usually try & avoid them, but in this case there are no techniques I know of that can get the inside (hidden) surface of the bra mesh to “follow” the surface of the figure during deformation well enough, and not at some time poke through the outer surface. It’s a classic virtual clothing problem. The meshes are too different to have them respond to the same armature exactly the same way, so it goes beyond careful weight painting, and I’m too far along in production to switch to something like a Mesh Deform modifier, which wasn’t available when I started the Kata project.

[Asano]

You could try applying the subsurf modifier and doing a render just as a test to see if that fixes the problem at all. Same with the triangulation (you should probably apply the subsurf before you switch to triangles).

I have a feeling that if you switch to triangles, it will either make it a little better, or little worse.

I have had problems with this in the past. I ended up pushing verts around until I could get them mesh’s to not intersect. Depending on the situation I might just try to blur out the object with DoF (which wouldn’t fly with this model seeing as how she is the focus).

EDIT: or you could render out the bra and the woman as seperate passes and composite them together. The masks created in the seperate passes will be just as jagged, but for the bra pass you could deform the woman mesh (the mask) slightly to make sure it doesn’t intersect. I know it would be a pain depending on what kind of shit it is, but at least that’s a solution.

[chipmasque]

@ Asano: Your suggestions sound like something that could work for a single-frame still render, but way too complex for animated frames. Unfortunately it’ll have to be a solution that doesn’t require special treatment of every rendered frame, or editing the mesh shape for every frame (i.e., shape keys). Shape keys might relieve the intersection problem, but would make getting a smooth animation very time-consuming to accomplish – I’ve learned that from doing Cloth sim -> shape key conversions, where essentially I have no other choice. Could be a fall-back technique, though, if nothing else seems to work.

I tried using FSA (Full Sample Antialiasing) but the results at the intersection were identical to the OSA version – it seems the problem is evident in every sample used to do the AA of either sort, again pointing at a geometry-related problem.

[Asano]

I know what you mean, it wouldn’t be very efficient for an animation.

From what I know about OSA, it averages the color of subpixels to come up with the color for the pixel. So it should fix the problem if you have enough OSA unless the values for that pixel are exactly evenly split between the bra and the woman. Did you have it all the way up to 16 when you tried raising it?
Have you tried rendering from a different angle or an a frame where the character is in a different pose?

[chipmasque]

@Asano

I know what you mean, it wouldn’t be very efficient for an animation.

From what I know about OSA, it averages the color of subpixels to come up with the color for the pixel. So it should fix the problem if you have enough OSA unless the values for that pixel are exactly evenly split between the bra and the woman. Did you have it all the way up to 16 when you tried raising it?
Have you tried rendering from a different angle or an a frame where the character is in a different pose?

My OSA was originally at 11, upping it to 16 made the intersection a little softer-looking but the overall raggedness stayed. Whatever is causing it is happening on a larger scale than AA processes. I’ve looked over the .exr files produce by using FSA (basically they are the individual samples used to produce AA) and the raggedness is there despite the changes in the “profile” of the pixels with each sample. That’s why I’m more and more certain it has something to do with the geometry. Even with render-time subsurf jacked up, the most ragged parts occur where the mesh lines (of the base, un-subsurfed mesh) meet the intersection. I’ll be trying some other things of a diagnostic nature later on, like applying the subsurf, or adding a multires level and seeing if that affects things. If it is geometry-related I might be able to minimize it that way for only the files/frames where it’s very visible, after all the animation’s finalized.

And from what I can see, the angle between the intersecting meshes had some effect on how bad the raggedness is, but camera and lighting angle don’t, except to make it more visible. But I can’t change camera angle and lighting that drastically just to cover it up.

[MadMesh (Niels Philipsen)]

I haven’'t been able to re-create the problem, even though it has happened to me. Could you post a simple blend that illustrates the problem? I’ve tried toe get it with 2 intersecting Suzanne’s with subsurf on and deformed, but it didnt work.

[chipmasque]

@MadMesh

. Could you post a simple blend that illustrates the problem?

I should be able to use the test file I’m working with now by simply applying the Armature and other modifiers to “freeze” things at that frame, then strip away all but the intersecting mesh area. If it is geometry-related, it should persist even through that kind of treatment. Materials and Lamps may need to all be the same, though, in case it’s related to those as well, though some of my other tests show it’s not. That will make this a big .blend file when packed because I’m using quite a few hi-rez texture maps.

BTW, I tried disabling all the textures for one test, still got the problem, and also when the intersection had the same Material on either side.

Like you, I’m not able to see this at all mesh intersections, and in fact this is the first time it’s been so pronounced, so it may be due to “special conditions” of this pose in terms of the relationship between the intersecting meshes. Identifying those conditions, or finding a way to fix the prob, will be of major use if later on it shows up again.

[Asano]

Have you tried using a different raytracer such as Yafaray?

[chipmasque]

@Asano

Have you tried using a different raytracer such as Yafaray?

I gotta chuckle a bit ironically at that suggestion – not that’s it’s not a good one, might help see where the problem is or isn’t, but unfortunately It’s not feasible at this point – too much of the look of the entire short is bound up in the Blender Internal’s capabilities and “style” of rendering.

I say ironically because I started out this project using the Yafray renderer, and really liked the results. But at one point Blender stopped supporting it in the same fashion, and it couldn’t do tangent space normal maps properly, so I had to switch to the BI – see the early pages of the Kata WIP thread (linked in my sig) if you want to see some early renders with Yafray. But that was before I started working with this costume, so this intersection issue never arose.

Might be something to test on a “theoretical” basis, though.

Anyway here’s a few more test results:

The subsurf has been applied to both meshes (level 1 on the figure, level 2 on the bra)

Here I’ve applied a level of multires as well, just to test it with higher rez meshes

Subsurf applied and the intersection sculpted to modify the intersecting angle – changes the look of the raggedness, lessens it, but doesn’t remove it. Still not as smooth as occluding mesh edges. This sort of imitates the shape key approach.

I’m outta ideas.

[chipmasque]

OK, geometry is NOT the problem:

Deciding to start from scratch, I stripped all the Materials off both meshes, back to the default that you get when first creating any kind of mesh. Voila, no raggedy andys.

Guess I’ll have to dissect my Materials/Shaders bit by bit now until I find the culprit.

Hope it won’t be something that’ll require re-rendering what I already have in the can, over 1000 frames, and many hours.

But at least there’s some hope of eliminating the prob.

[Asano]

You said you removed all of the textures when you did a test earlier, So then I guess you’ve narrowed it down to Materials. If that’s the case, at least you can use all of your textures still. Try putting the materials for just the bra back. if it’s still not jagged then try just the woman. Then you’ll know if it’s the combination of the materials that’s causing the problem.

[chipmasque]

Found the problem.

Blendgremlens!

You know, those little orange, white and blue meanies that sometimes infest a .blend file and make it screw up for no good reason? They visited me last night, and were likely responsible for making me miss the real reason – MadMesh, you get a cookie – it was (ta-daa)… ZTrans. :eek:

“But,” you say somewhat heatedly, “you said you tested ZTrans!”

Did. Last night. Toggled it on both Materials, separately and together. No luck fixing the raggedies. Also tried the ID mask thing. Lots of other stuff. Gave up. Decided to do something I’d been putting off for a while, adding a little touch-up to the face texture around the eyebrows, to make 'em less artificial-looking. Simple paint job. Decided to save the new texture as a 4096 PNG since PNG’s so damn efficient (was using TGA).

Eyebrows looked fine. Then I enabled the layer with the particle hair and rendered again to check the color match, etc. The hair Material had gone haywire. Dark brown color. Nearly transparent. Completely different than anything I’d ever rendered before. Yet I hadn’t touched the hair Material, it’s totally separate from the figure mesh Materials, including the Eyebrows Material. Nothing in common, like a texture.

I checked all the hair specs. Not a hair out of place (so to speak). Everything as was, as s’posed to be. Rendered again. Hair still f***d up. Close file, no save. Pick it up again in the morning.

Blendgremlens. Likely screwed up the ZTrans tests as well. At least, I can’t think of any other reason why I got different results today. ZTrans was the last Material option that was different between my Materials and the default, and it was the last tested. But it was the right one.

Blendgremlens.

Hate 'em.

PS: Thanks, big thanks, to everyone who tried to help. Much 'preciated.