I’ve recently looked into Vertex Painting in blender for ID masks for texturing. It seems like it would be a nice option instead of always assigning different materials in various colors then baking a diffuse map. However, I always seem to get some pretty intense aliasing whenever I use vertex data.
Baking the vertex colors in substance painter seems to yield the same thing as well.
Is there anyway to get a completely constant separation between colors on a single mesh?
I assume this is due to your mesh verts beeing ripped as in the verts are not connected. Its like not having a water tight model as the verts are not welded. By default when you assign vertex colors they would blend but since you assigned it to a face mask instead i presume it ripped the verts when it crosses. It would explain the aliasing and is usually the limitation of using vertex colors for hard face selections.
You don’t actually want anti-aliasing for ID maps. They can’t be anti-aliased. Either something has an ID of one value or another, but nothing in between.
When you create actual masks from an ID map, that’s where the software may be able to soften it out, see this.
It’s not separated, no. All one solid mesh.
Yea, you want your ID masks to be clean obviously, I just don’t understand why it’s being applied like that. I am highlighting the faces I want to color in edit mode then using shift + K to fill with a solid color in vertex paint mode. This should technically give a clean result as the area is clearly define, but it still gives this jagged look. Having the model be subdivided or not doesn’t seem to affect the result either.
If you’re talking about the viewport display, they’re aliased because vertex colors are based on geometry and you don’t have geometric anti-aliasing enabled for your viewport (System > Window Draw Method > MultiSample).
Thanks for following along so far I appreciate it!
I’m aware that the viewport is going to be aliasing things unless you tell it not to, the problem arises when the vertex color data is not holding up those edges in other software (which it should I think). if I take the vertex color from the attribute node and plug it into a diffuse, it will display jagged in the viewport, but if I render it it’s smooth. Yet! if I bake the vertex data in substance painter it still gives a jagged result. I’ve been baking in 2048 and higher, but it still is jagged. I exported a diffuse map from substance and applied it in blender then hit render to see if it cleaned up, unfortunately it’s worse!
When you bake it to a texture, you want it to be jagged, you don’t want it to be smooth. Smoothness implies blending between the colors, but you don’t want that. You want one pixel to be either one color, or another, nothing in between. That’s what I mean by “you don’t want anti-aliased ID maps”. It’s the only way to preserve the ID unambiguously.
Click on the link I gave in my initial reply. How you deal with the fact that ID maps are aliased/jaggy depends on the software using the map.
To solve your particular problem (i.e. that sphere), just make sure the UVs are fully covered with one ID color and have the UV islands take care of the boundary.
I think we are getting stuck on semantics here. The transition from one Color to another in an ID mask should be “SHARP”. Any color blending leads to one a fuzzy non defined edge, we both know this. What I was trying to get across is that there is jaggedness in the transition from one of my colors to another. Jagged is still “SHARP”, which is what I want, but it is also zig-zagged, bleeding a bit into the each others color space. You should clearly be able to see this in the image with the yellow and red sphere I had in substance. You can even see the UV’s where the islands have a bit of each others color JUST around the border.
The link you gave me was very helpful, I learned that in substance with the help of some filters you can blur this “zigzag” effect then use either levels or histogram to try to level them out, creating a slightly blurred, but “smoother” transition! So thank you for linking that !
My only remaining question is that, from my current understanding, it seems like there is simply no way to avoid that zig-zagged effect? Simply because the way softwares deal with color is with pixels, and creating a perfectly sharp edge is not really possible due to a number of issues. So the best we can do is merely treat the outcome, with filters, levels etc.
Is that right? (Really trying not to be rude, just really want to understand the basics here)
Yes, it’s difficult to talk about. In your particular case, judging from the substance painter image you posted, the problem is that the UVs are not fully covered by a single ID.
Your UVs should have some margin so that the ID color can “bleed over” and no matter where the triangle is sampled, it will only get one ID. Blender has a “Margin” bake setting to extend pixels automatically, to get that “bleed over” effect. This is a general treatment for UV seams, not related just to IDs.
That way, the geometry (UV seams) will take care of the separation and it will be perfectly sharp and not zig-zagged.
In the general case, where the UV seams do not necessarily coincide with the ID boundaries, yes, you can get some artifacts, but it depends a lot on the software and process you are using. It’s good to sometimes “zoom out”, look at the big picture and realize that even professional artwork is never perfect.
Oh, that’s new for me. The UV margin is something I’ve been wondering if there was a setting for. And the baking margin setting just controls how much it bleeds over in baking diffuse or other maps. Good to know. Funny thing is, even when I gave a massive distance between the islands and manually painted two blocks of color in photoshop, with zero chance of overlapping near the UV islands and use that as a color ID map, substance still read it with zigzags. The only way I can seem to get it clean and sharp is by have two separate materials assigned. Alas, I guess vertex painting isn’t the way to go for these things.
