Softening vertex color AO in Python?

Hi,

Currently when Blender bakes real ambient occlusion to vertex colors they come in with sharp edges between faces. These transitions curiously get smoothed if one uses blur tool anywhere in the vertex paint viewport (I guess by combining all vertices in same location?). How do I access this feature from python? Trying to use stroke from script makes Blender to complain about incorrect context.

i don’t believe you can fake the whole context to simulate brush strokes, but you can access the vertex color loops and change the color, with a few formulas it should be possible to smooth them.

Okay, what I want to do is to combine vertex colors for vertices that are in the same location (so that a single vertex has a single color). Currently baking to vertices splits vertices based on faces creating hard edges without soft transitions. I’m not sure but this may be a bug in Blender vertex baking as a click anywhere in viewport with blur tool combines these splitted vertices into one color.

Ah well, I have to see if I somehow manage to iterate vertices so that they get combined.

Vertex Colors are actually not Vertex Colors but Loop Colors. Vertices can be shared among faces, loops not. If you just want to mix the colors of all loops with the same vertex, that would be really easy to script.

Okay, I figured it was something like that. I made key-value dictionary with vertex indices and their wanted color but how can I know which loop index is what vertex index? In other words, how can I set vertex colors per actual vertex and not by loops?

Btw, does FBX support this Blender structure of loop colors instead of vertex colors? I would think that hard color edges get split and object vertex count would raise in game engine / OpenGL?

I don’t know if FBX supports per-face vertex colors, otherwise you would need to split verts of course. Or use an interpolation of the colors, or pick just the first vertex color (rather ugly solution).

I wrote this operator to calc the average color:

import bpyfrom mathutils import Color


def avg_color(color_list, ignore_white=False):


    color = Color()
    len_list = len(color_list)
    white = Color((1,1,1))
    all_white = True
    
    for col in color_list:
        if ignore_white and col == white:
            len_list -= 1
        else:
            all_white = False
            color += col
    if ignore_white and all_white:
        color = white
    elif len_list != 0:
        color /= len_list
    return color


def main(self, context):


    vcol_map = {}
    me = context.object.data
    vcol_data = me.vertex_colors.active.data


    for l in me.loops:
        col = vcol_data[l.index].color
        try:
            vcol_map[l.vertex_index].append(col)
        except KeyError:
            vcol_map[l.vertex_index] = [col]
            
    
    for i, l in enumerate(me.loops):
        vcol_data[i].color = avg_color(vcol_map[l.vertex_index], self.ignore_white)
        
    me.update()
    


class MESH_OT_vertex_color_loop_average(bpy.types.Operator):
    """Calculate the average color of shared vertices' loops"""
    bl_idname = "mesh.vertex_color_loop_average"
    bl_label = "Vertex Color Loop Average"
    bl_options = {'REGISTER', 'UNDO'}
    
    ignore_white = bpy.props.BoolProperty(
        name="Ignore white",
        description="Ignore white (prefer colors, only use white if all loops are white)",
        default=True
    )
        


    @classmethod
    def poll(cls, context):
        return (context.object is not None and
                context.object.type == 'MESH' and
                context.object.data.vertex_colors.active is not None)


    def execute(self, context):
        main(self, context)
        return {'FINISHED'}




def register():
    bpy.utils.register_class(MESH_OT_vertex_color_loop_average)




def unregister():
    bpy.utils.unregister_class(MESH_OT_vertex_color_loop_average)




if __name__ == "__main__":
    register()

Very classy! This should come built-in with vertex color baker. I noticed the script has to be run twice to smooth all the vertices, otherwise some edges will stay sharp.

Now I just have to figure out what you’re doing to implement similar thing in my own script…

FBX supports hard edged vertex colors (checked visually in Unity3d) but the thing that worries me is the OpenGL end as each vertex can only have one color and if there are more colors the vertices will be split (AFAIK). If Blender outputs only per face vertex colors then each vertex is split and the GPU has to render more vertices which might cause performance drop on low-end hardware.

edit:
Checked and Blender exports things very properly. Vertices which are smoothed are smooth and vertices which are hard are split, so no degenerate vertices are generated unnecessarily.

Great, but i wonder why you have to run my operator twice… It seemed to do it in a single run. Can you provide a test setup (.blend)?

Here is one quick test file. First object shows plain ao bake, second loop average done once and third loop average done thrice (even two times was not enough for all the vertices to be smoothed).

edit:
I’m not sure, but maybe one should iterate individual vertices, check into which loop they belong, take average on all those different colors on each loop and set that average back into each loop.

Attachments

vertex_loop_average_example.blend (498 KB)

one should iterate individual vertices, check into which loop they belong, take average on all those different colors on each loop and set that average back into each loop

That is exactly what my script should do! I got the error fixed, but I honestly don’t have a clue why avg_color returned different colors for the same list of colors… Anyway, it should be a bit faster now as well and you only need to run it once.

For the example .blend you posted, you may uncheck the “Ignore white” property in the Tool props panel (T key in view 3d, at the bottom)

import bpy
from mathutils import Color

def avg_color(color_list, ignore_white=False):
    color = Color()
    len_list = len(color_list)
    white = Color((1,1,1))
    all_white = True
    
    for col in color_list:
        if ignore_white and col == white:
            len_list -= 1
        else:
            all_white = False
            color += col
    if ignore_white and all_white:
        color = white
    elif len_list != 0:
        color /= len_list
    #print("ret color", color)
    return color

def main(self, context):

    vcol_map = {}
    me = context.object.data
    vcol_data = me.vertex_colors.active.data

    for l in me.loops:
        col = vcol_data[l.index].color
        try:
            vcol_map[l.vertex_index].append(col)
        except KeyError:
            vcol_map[l.vertex_index] = [col]
            
    for key in vcol_map:
        vcol_map[key] = avg_color(vcol_map[key], self.ignore_white)
    
    for i, l in enumerate(me.loops):
        vcol_data[l.index].color = vcol_map[l.vertex_index]
        
    me.update()

class MESH_OT_vertex_color_loop_average(bpy.types.Operator):
    """Calculate the average color of shared vertices' loops"""
    bl_idname = "mesh.vertex_color_loop_average"
    bl_label = "Vertex Color Loop Average"
    bl_options = {'REGISTER', 'UNDO'}
    
    ignore_white = bpy.props.BoolProperty(
        name="Ignore white",
        description="Ignore white (prefer colors, only use white if all loops are white)",
        default=True
    )

    @classmethod
    def poll(cls, context):
        return (context.object is not None and
                context.object.type == 'MESH' and
                context.object.data.vertex_colors.active is not None)

    def execute(self, context):
        main(self, context)
        return {'FINISHED'}

def register():
    bpy.utils.register_class(MESH_OT_vertex_color_loop_average)

def unregister():
    bpy.utils.unregister_class(MESH_OT_vertex_color_loop_average)

if __name__ == "__main__":
    register()

Works well, thanks a bunch!

Too bad, i can’t add this script to contrib SVN - there will be a more advanced C-operator for exactly this in 2.67! It will support face selections, so you don’t need to smooth everything.

Oh well. This code should still be useful for some people, as min/max operators are easily switched with average.

Will this C-operator be available from Python API? Smoothing AO is just a part of my own script.

all operators are, some require certain contexts - but since we can override them or use tricks, we can use almost all of them (except e.g. lasso selection, i don’t think we can control this from python)

it is going to be
bpy.ops.paint.vertex_color_smooth()