WIP: Camera Face Tracking

So I’ve repeatedly talked a big talk about doing face-cap with blender’s new video tracker, but shy of bolting a web cam to my face or actually doing code, I wasn’t going anywhere fast, but today I finally got draft zero to run, and here it is:


import bpy

FirstFrame=bpy.data.scenes[0].frame_start
LastFrame=bpy.data.scenes[0].frame_end

i=FirstFrame

#Tracker Handles
Chin=bpy.data.movieclips[0].tracking.tracks['chin']
LipUp=bpy.data.movieclips[0].tracking.tracks['lipupper']
WLeft=bpy.data.movieclips[0].tracking.tracks['W_Left']
WRight=bpy.data.movieclips[0].tracking.tracks['W_Right']

#Shape Key Handles
OpenMouth=bpy.data.objects['Plane'].data.shape_keys.key_blocks['m_open']
NarrowMouth=bpy.data.objects['Plane'].data.shape_keys.key_blocks['m_narrow']

while i < LastFrame:
    i=i+1
    
    bpy.ops.anim.change_frame(frame=i)
    
    OpenMouth.value=Chin.markers[i].co.y-LipUp.markers[i].co.y
    OpenMouth.keyframe_insert('value',-1,i,'mouthopen')
    
    NarrowMouth.value=WLeft.markers[i].co.x-WRight.markers[i].co.x
    NarrowMouth.keyframe_insert('value',-1,i,'mouthnarrow')

By isolating out distance mathematically, I at least get rid of the worst of the effects of moving my head, although if I turn too far the markers won’t stick so well. To do list:

  • more complicated math for better tracking
  • normalization built into the code instead of doing it by hand afterwards
  • track things other than just mouth, easy enough but should have normalization working first so I don’t have to scale 10+ curves to fit after running the script
  • A nice interface
    And finally,
  • Plugginification.

It rips tracks from the tracker straight to the shape keys, cutting out a whole heap of steps from the follow-empty method, which when this script is done should get a lot better performance too thanks to things like calculating distance and normalizing the overall scale against forward-and-back motion.

So far the results actually haven’t impressed me that much, but my start footage is pretty questionable, so we’ll see…