Tips for those using NVIDIA's Audio2Face with Blender

For those using Maya, the A2F workflow is easier, since A2F can export a Maya Cache and can also convert the animated mesh into Maya blendshape animations.

For us folks in Blender, there is an addon that can import Maya Cache into Blender as a shapekey animation, explained in this video. While it works, I worry the addon might not be kept updated forever, so I attempted to work with the native USD file.

First issue that arose was that you can’t use shapekeys on top of the Mesh Sequence Cache modifier reading the USD file, which severely limits your ability to tweak the A2F animation, except with bones and an armature, which tend not to be sufficiently detailed for realistic facial animations.

Second issue is that you need to tick the Subdivision checkbox in Blender when importing the USD file, or else the mesh will be a simplified version of what you see in A2F. Ticking the option adds a Subdivision Surface modifier on top of the Mesh Sequence Cache, which brings the detail back in, but at the cost of a much higher polycount.

I found a rather obscure workaround to both of these issues in Blender 3.2.1, however, so thought of sharing it here.

First, activate a built in addon called “NewTek MDD format”. Import the USD file into Blender with subdivision turned off. I only tested with the continuous skin mesh inside the USD file, so you might want to avoid exporting anything else with this file out of A2F. Next, select the mesh and export as Lightwave Point Cache (.mdd). Duplicate the mesh, exclude the Mesh Sequence Cache modifier from the copy and, with it selected, import the mdd file back in. this will convert the USD animation read through the MSC modifier to a shapekey animation. For whatever magic reason, this animation preserves the details seen in A2F, even with a lower polycount target, while allowing for additional shapekeys to be added to the resulting animation.


Right: original USD cache read through the MSC modifier on a 23k vertices bust, no subdivision.
Center: shapekey animation generated by importing mdd file on the 23k vertices bust, no subdivision.
Left: original USD cache read through the MSC modifier on a 143k vertices bust, x1 subdivision.

Furthermore, I will be investigating how to best handle the various additional meshes included in the human face, such as eyes, eyebrows, beard, etc, and post results in the comments. The same video mentioned above also describes a workflow for these attachments with the Mesh Data Transfer addon.

3 Likes

Have you seen better results with the new V2 of Audio2Face?