Getting OpenPose Mocap output into Blender

OpenPose seems to have the best performance of any monocular AI mocap. The only problem is that it does not have .bvh output unless something like Forth MocapNet is used to make the conversion, which does not work for fingers yet.

For someone who is familiar with python programming in Blender it might be easy to take the 2D output from OpenPose to create joint movements in pose mode, with the same camera lens, distance and orientation with the subject as the original capture video. An OpenPose compatible rig is available for Blender from MakeHuman. This is an interesting approach to convert any character movement video into a Blender armature action that would make a great popular plugin. Anyone interested?

1 Like

I used Mediapipe to get face mesh before. It also provides pose detection and a decent Python interface. Have you looked into it?

Just learned about Mediapipe today. Carlos Dubarreto has python code to import Mediapipe into EasyMocap, but adapting that is a bit over my head. Please see if interested: https://github.com/carlosedubarreto/b3d_mocap_import/issues/10#issuecomment-855189019

Do you know if body and hand recognition apps based on Mediapipe are available? As a non-programmer OpenPose works for me now, though Mediapipe could be better in the future. The approach I am suggesting seems fairly direct and could work as a standard Blender plugin.

A continuation of this topic of importing AI mocap into blender, but using mediapipe is at Mediapipe for Blender - python programming

After looking into MediaPipe it seems there is a basic limitation of accuracy with that approach due to a lack of maintaining a set length for each bone. So the initial approach suggested in this thread is being developed at https://github.com/Aurosutru/Blender-plugin-for-OpenPose-import/tree/main
At present a rough outline is available. Code contributions are most welcome.

More work on getting MediaPipe running with Blender is going on at https://www.patreon.com/posts/dev-code-preview-53279931

The developer is hopeful this will work well. You can download a .blend file there and with some slight modification give it a try.

2 Likes

Thanks for sharing the link @Aurosutru
Today was released a new version with a better mediapipe data import and even a menu to make the loading of video easier.
As a bonus, there is also a vroid rigged model and remap files to use with rokoko retargeting solution or autorigpro.

Hope you like it.
https://www.patreon.com/posts/dev-preview-and-53711794

There is a video tutorial showing how to use it