bRigNet - neural rigging for blender

Hi guys,

I have made an addon that uses RigNet to predict Armature and weights. It’s still experimental but it kinda works

you can find the code and instructions here

their predictive model works better with a T-pose, I haven’t tried to train a custom model yet

Have fun!

31 Likes

Impressive! Thanks for sharing. Would be great to see this integrated in Blender.

6 Likes

Interesting. I’ll give it a try. :cupcake: :cupcake: :cupcake:

1 Like

Very cool!

Had a quick glance at the code. Is it correct that the inference is run in PyTorch?

1 Like

Hi guys!

@Metin_Seven, you mean as an official addon? At the moment it’s very unlikely: too many dependencies, not cross platform, and still not mature enough. But there is hope

@kabu, oh yes please, make yourself at home

@DeepBlender, I am totally new to neural network and AI, so far I have only converted their quickstart script in order to have RigNet talk with blender. You are correct: they use PyTorch. Is it a good framework for neural networks in blender? Do you have any suggestion towards a more direct implementation, perhaps having the model learn from .blend files instead? I haven’t touched the learning part yet

thank you all!

3 Likes

PyTorch is a great framework for the training of neural networks, alongside with TensorFlow. Both aren’t well suited to deploy trained networks for Blender or standalone applications in my opinion. Something like ONNX(Runtime) is better suited for this, as it allows you to easily load a neural network and execute it on CPU/GPU/… . There are converters from PyTorch/TensorFlow to ONNX, but not all operators are supported. I am currently extensively testing the ONNXRuntime and it looks very promising.
When it comes to train neural networks, the bottleneck is very often the loading/augmentation of the training data. That’s why I wouldn’t recommend to load training data from Blender directly, but rather to save the data in files that can directly be loaded for the training.

2 Likes

Holy crap, this is damn interesting !

I tried Installing this but after I create the rignet environment folder I have this error

Python: Traceback (most recent call last):
File “C:\Users\Animator-Ed\AppData\Roaming\Blender Foundation\Blender\2.91\scripts\addons\brignet-main\brignet.py”, line 28, in execute
from . import rignetconnect
File “C:\Users\Animator-Ed\AppData\Roaming\Blender Foundation\Blender\2.91\scripts\addons\brignet-main\rignetconnect.py”, line 10, in
import torch
File “C:\Program Files\Blender Foundation\blender-2.91.0-31\2.91\python\lib\site-packages\torch_init_.py”, line 127, in
raise err
OSError: [WinError 126] The specified module could not be found. Error loading “C:\Program Files\Blender Foundation\blender-2.91.0-31\2.91\python\lib\site-packages\torch\lib\caffe2_detectron_ops_gpu.dll” or one of its dependencies.

location: :-1

I even copy the lip site package files from the rignet env to blenders python but nothing it fails again I am thinking maybe a missmatch of python versions and the compiled windows RTree files but this installation is too hard

I have tried all I can following the exact steps as in the git and setting the environment for Rignet but I get this error

Python: Traceback (most recent call last):
File “C:\Users\Animator-Ed\AppData\Roaming\Blender Foundation\Blender\2.91\scripts\addons\brignet-main\brignet.py”, line 28, in execute
from . import rignetconnect
File “C:\Users\Animator-Ed\AppData\Roaming\Blender Foundation\Blender\2.91\scripts\addons\brignet-main\rignetconnect.py”, line 10, in
import torch
File “D:\rignet\Lib\site-packages\torch_init_.py”, line 127, in
raise err
OSError: [WinError 126] The specified module could not be found. Error loading “D:\rignet\Lib\site-packages\torch\lib\caffe2_detectron_ops_gpu.dll” or one of its dependencies.

location: :-1

I would appreciate if you can point me to what it might be happening

Thank you.

It requires PyTorch to be installed. The way to go is most likely to install pip for Blender’s Python and to install PyTorch for it (using pip). Be prepared to handle with PyTorch issues.
Don’t have time to give you appropriate links, but what I described should be sufficient to google for them.

1 Like

Thanks but I when intalling the enviroment for rignet I intall pychart with this command
conda install pytorch==1.6.0 torchvision==0.7.0 cudatoolkit=10.1 -c pytorch

anyway I went and intall it for blender’s python too using

pip install torch==1.7.0+cu101 torchvision==0.8.1+cu101 torchaudio===0.7.0 -f https://download.pytorch.org/whl/torch_stable.html

but got more errors so I don’t know is just to difficult for me but defo pytorch is installed in both places now

Unless it needs CUDA since my GPU is AMD 480rx I don’t think it runs cuda but I also tried installing the cuda version but on this test it says is not runing image

Hi @ImpactFrames,

though the code seems to use non-cuda hardware for some operations, AFAIK RigNet is using the cuda framework.

The install can be painful, with lot of dependencies that can also vary according to the hardware and system: at the moment there’s no way around pip or venv/conda.

My opinion as a novice in this field: AI is just becoming to circulate among the masses (such as myself), and only a fraction of the hardware/software park is ready to host AI implementations

Hopefully this is going to change soon

1 Like

Yes, I was thinking that since my computer might be too different from your configuration even if I manage to install it the results might take longer or be different from yours.

Right now I am installing all the dependencies without cuda using just cpu like in this guide https://github.com/rusty1s/pytorch_geometric

apparently I also have to set this path ‘C:\Program Files\Blender Foundation\blender-2.91.0-31\2.91\python\Scripts’ to the pythonpath if I don’t manage by today I will probably desist of the idea

BTW as a total newbie and having no idea what I am doing still it makes sense so far and having the computer do some of the rigging would be worth it for my workflow.

Thanks.

I went installed an old gforce 970gtx NVIDIA card with cuda I had on another can. Reinstalled the rignet environment again but since the python shipped with miniconda is either 3.8 or 2.7 it only let me install this wheel Rtree-0.9.4-cp38-cp38-win_amd64.whl since the pip install Rtree‑0.9.4‑cp37‑cp37m‑win_amd64.whl says it’s not available for my system despite of being also in the folder anyways I installed 3.8 then this line failed

pip install numpy scipy matplotlib tensorboard open3d==0.9.0 opencv-python

so I used

pip install numpy scipy matplotlib tensorboard open3d==0.10.0 opencv-python

instead and continued with all the other dependecies as the rignet environment guide

installed pytorch with cuda for the blender’s python and upgraded pillow as in your directions then pointed the rignet enviroment folder in blender and added blender python scripts to my sytem enviroment variable paths

result blender throws this beautiful error
image

Now I give up.

As mentioned previously, that’s very unfortunate, but not surprising… That’s literally the reason why I am looking into ONNX (+ Runtime), which is quite simple to install (at least the CPU version, but even the GPU one is relatively painless). I am planning to convert this model into ONNX in the future, but I don’t have the time at the moment.

1 Like

Yes is like the minimal variation of package version and hardware configuration trows all off board but I recognize @pKrime efforts on bringing this to blender and I expect everyone else would also welcome your efforts streamlining this installation. via ONNx I am a noob but also doesn’t help that my hardware is always 3 or more gens behind, I guess other people may have an easier time than me installing it.

Hello everyone, sorry if this out of places, in meantime i try to understand whats wrong here, after finally make pytorch detected in blender, i’m running to new issues

Traceback (most recent call last):
  File "D:\Aye\blend\2.90\scripts\addons\brignet-main\brignet.py", line 28, in execute
    from . import rignetconnect
  File "D:\Aye\blend\2.90\scripts\addons\brignet-main\rignetconnect.py", line 11, in <module>
    from torch_geometric.data import Data
  File "D:\Aye\Lib\site-packages\torch_geometric\__init__.py", line 2, in <module>
    import torch_geometric.nn
  File "D:\Aye\Lib\site-packages\torch_geometric\nn\__init__.py", line 2, in <module>
    from .data_parallel import DataParallel
  File "D:\Aye\Lib\site-packages\torch_geometric\nn\data_parallel.py", line 5, in <module>
    from torch_geometric.data import Batch
  File "D:\Aye\Lib\site-packages\torch_geometric\data\__init__.py", line 1, in <module>
    from .data import Data
  File "D:\Aye\Lib\site-packages\torch_geometric\data\data.py", line 8, in <module>
    from torch_sparse import coalesce, SparseTensor
  File "D:\Aye\Lib\site-packages\torch_sparse\__init__.py", line 13, in <module>
    library, [osp.dirname(__file__)]).origin)
  File "D:\Aye\blend\2.90\python\lib\site-packages\torch\_ops.py", line 105, in load_library
    ctypes.CDLL(path)
  File "D:\Aye\blend\2.90\python\lib\ctypes\__init__.py", line 364, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: [WinError 126] The specified module could not be found

location: <unknown location>:-1


Is this means something that i not installed yet or could be another thing ?
Thank you for your attention

Hi guys,

sorry that it can be painful to make it work. If it wasn’t for all the insights that I have received on how to improve it, I would regret having released it so soon

to recap:

  • at the moment the model uses CUDA and pytorch, but as noted by @DeepBlender there are ways to convert to lighter frameworks: we could even manage to use just numpy, albeit I am not sure of how it would perform
  • there are ton of other dependencies that would better go: we should not need all those 3d libraries inside a 3d application

Unfortunately it’s a lot of work, so the friendly release might take some time

ah thank you i finally make it works !!!

1 Like