Undo operator that runs in edit mode and modifies object transform matrix

Hi, I am working on a set of operators that modifies the object matrix based on selected elements of a mesh: https://github.com/eliemichel/AlignTools

Basically, I compute a transform matrix from the user selection, then apply it to the mesh data and its inverse to the object transform. So it has to be an operator running in edit mode. But at the same time it also perform changes relative to object mode.

Here is a simplified version, that just translates the origin:

class Test1Op(bpy.types.Operator):
    bl_idname = "transform.test1"
    bl_label = "Test1"

    @classmethod
    def poll(cls, context):
        # Check that we run in edit mode
        return context.active_object.mode == 'EDIT' and context.object.type == 'MESH'

    def execute(self, context):
        # Get some arbitrary transform from edit mode selection
        tr = Matrix.Translation([0, 0, 1])

        bpy.ops.object.mode_set(mode='OBJECT')
        obj = context.object
        mat = tr * obj.matrix_world
        
        # Apply the opposite transform to the mesh data
        mesh = obj.data
        mesh.transform(tr.inverted())
        mesh.update()

        # Set new object matrix
        obj.matrix_world = mat
        bpy.ops.object.mode_set(mode='EDIT')
        return {'FINISHED'}

I tried to versions, this one and another one that does not temporarily switch mode and uses bmesh to transform the mesh. Both show the same symptoms.

When undoing this operator, only the mesh transform is inverted.

So, while the operator is made such that the world position of the vertice does not change, undoing makes everything move.

I tried with and without bl_options = {'REGISTER', 'UNDO'}. Ultimately I need it because I want to add properties. For now, it is terrible if I add for instance a fac = FloatProperty(name="Factor", default=1.0) and use it in the matrix, like tr = Matrix.Translation([0, 0, self.fac]), then the redo completely messes up with the mesh.

I believe the problem comes from the fact that undo stacks are different from object and edit mode, but how can I get this around?

1 Like