Animation - Music Sync

(Aschenstern) #1

Hello fellow blenders,

I know this question has been probably asked a ton of times. But the thing is : even after soing some extent research i didnt really find a good solution so i wanted to ask you guys, maybe i misunderstood something.

So i have prepared some scenes and own rigged models to do a music video. Basically i have a song and i want to animate the models to it. The tricky part is: i want them to syn exactly to the music/beat.
For example : lip sync for singing but also mathed movements for example hand clapping etc.

I tried using the audio scrub method. But for me it didnt really work out : even with waveforms and scrubbing i cant really make our any word sang in the song , not even speaking about my animation being syncronised with the music.

So , does anyone know a good way to do it? Maybe i misunderstood something?
Because i cant imagine how to do a good synced animation by using such methods.

Thank you so much!

(Mousy) #2

Use pose library for reusable set of phoneme poses and use them. It is extremely difficult to separate voice from music for automation. This requires a well-trained neural network. Therefore the animation of singing should be done manually. It’s difficult, but it’s possible.

For animation of dance, you can record movement of some bones with the mouse, then smooth or simplify curves, then manually do adjustments, focusing on the recorded movements of the parts of the rig

You can use the NLA editor to assemble a complex dance from simple actions.

(Clockmender) #3

See this thread:

And this one:

Then look at my Youtube channel (in my signature at the bottom of this post) at some of the latest vids, then let me know if you want to take this approach further. it requires a MIDI file to animate exactly to the music. I have my Human model tapping one of her feet to the beat now, ready for a later video of the “Clockophone”.

I have written some MIDI Sync nodes as an add-on top Jacques excellent Animation Nodes.

Cheers, Clock.