Mediapipe for Blender - python programming

Mediapipe is impressive. It is much faster than OpenPose and Wrnch and usually self-reports 100% accuracy, even for hands. It even runs on smartphones. For an example of python programming with Mediapipe see: https://www.youtube.com/watch?v=EgjwKM3KzGU

A good start on a Mediapipe plugin for Blender is at https://github.com/zonkosoft/BlendyPose This plugin worked well with Blender 2.92 but has a problem with changes made in 2.93. The developer is asking for contributions to the code.

Python code for Mediapipe import via another Blender plugin is at https://github.com/carlosedubarreto/b3d_mocap_import/files/6602077/ESQUELETOOK_mediapipe_blender_v1.zip

It seems that with adaptation this code, which calculates x, y, and z bone information, could take the landmark data from BlendyPose and convert it into a .bvh file for easy use in Blender.

More work on getting MediaPipe working with Blender is going on at https://www.patreon.com/posts/dev-code-preview-53279931

The developer is hopeful this will work well.

2 Likes

Thanks for sharing the link @Aurosutru
Today was released a new version with a better mediapipe data import and even a menu to make the loading of video easier.
As a bonus, there is also a vroid rigged model and remap files to use with rokoko retargeting solution or autorigpro.

Hope you like it.
https://www.patreon.com/posts/dev-preview-and-53711794

There is a video tutorial showing how to use it

1 Like

Thanks for your work Carlos! I am wondering, is your work open sourced? I am currently starting some research for an “avialable for everyone” solution to capture signs for sign language and I find your work very interesting. I’d be happy to contribute if I can.

Have a nice day.

1 Like

Thanks for the words.
It is open source.
The main code is in this link

The other links here on this post, are from some tests I’m doing before adding to mocap_import (the full addon)

I never worked with somebody helping on the code, and to be honest I dont know how to deal with some other things, like, I receive donations for the addon on gumroad and patreon, and I dont think its fair for me to receive for somebody else works and neither I have an idea of how to split…

I didnt think somebody would want to be part of it :sweat_smile:

1 Like

Nice to hear it’s on github! I am not interested in money so don’t worry about that. This is some research I am doing, so it makes sense to me to contribute to work already being done instead of reinventing the wheel. Not that I am an expert in animation anyway.

I’ll be experimenting with your plugins for a few days and then I’ll try to contribute if I can. Thanks again!

1 Like

Thanks a lot!!!.
And also if you have suggestion, or if you have som idea to implement i can give some directions and codes to help on what you want to archieve.

This is a great achievement. Am not using rokoko or autorigpro, but will look into open source tools to transfer the animation from the vroid rig to a MakeHuman rig.

Do the 4-5 BlendyPose code lines for smoothing look useful?

1 Like

Those 4 lines of code helped a lot.
It gave me better depth position and it showed me an very simple way to put the animation in the right axis (flipping 2 axis)

Can the output be made smoother? Significant jerking is seen in the Michele video you posted and in results from my test input videos, though the video motions themselves are smooth. Can the smoothing code found in BlendyPose be incorporated? It works well there.