Auto-lipsync and blender ?

Hello,

I’m looking for a quick and dirty, fully automated lipsync solution that I can use for animation with blender.
What I’d like to do is to use a .wav / sound file as an input and get something that I can work with ( ex: a text with some phoneme / time information, or some animation curve data).

I’ve tried to import a .wav inside blender and convert it to a F-Curve, then mix it with some random animated shape keys, but I was wondering if there isn’t a slightly better solution :confused:

My project is a cartoony / basic style animated serie, so having a not-so accurate motion can work, It’s quantity over quality here :slight_smile:

I’ve found about a software called Smartbody, but maybe it’s a bit overkill , I haven’t succeeded in compiling it anyway . Maybe I should look more into it.

There is also FaceFX, but a bit too expensive/overkill for what I’m planning.

Papagayo / Jlipsync : I’ve tried them, I’m looking into something where I don’t need to type text and tweak phonemes.

So have you got any clues about some piece of software or python library that can be used ? Or , some tutorial, general informations about the subject ? How would you do it ?

I can do some python scripting, if I manage to make something that work I can release my work as an addon or share my results.

Thanks !

I don’t know much about coding and stuff, but if you really only want basic mouth flapping maybe something with animation nodes could work-driving a jawbone or something.

With the amount of work I’d have to do to find a shortcut, I’d sooner just animate the mouth quickly. At a basic mouth movement level, it’s not the hardest bit to animate. Much harder to do a good walk :wink:

Maybe someone know an addon for it, but basically i think what you want, is a lips synchronisation, animation tools, and then load the animation in Blender… Need to see what software ( free, there should be one or two free ) can work with the animation import with Blender.

I remember there was a free alternative to faceFX ( the soft was developped in collaboration with AMD, if my memory is good ). Now does it still exist ? or have become paid… i dont know.

Some camera “sync” face tools are free ( record a face with the webcam, and then it animate the CG face, problem is you will need to “play” the text yourself… )

The problem is the “good software” are offtly expensive as developped by or for animation studios.

Note for later: this could be a seriously good addon for Blender to get the possibile to animate directly the face and sync lips for animation purpose on this way ( i know 3DSmax and Maya had some ( pay ) addon for this … )

Thanks guys !

@Hammers :
Indeed ! it’s worth trying animation nodes, thanks for remind me of that great addon !
At least it can help me to prototype something. I hope to find something a little bit “clever” that can find some phonemes rather than just up/down moves of the jaw. At least having some kind of “O” “E” “A” shapes for the mouth more or less synchronized with the sound.

As I will have to do a lot of lips sync with several voices it’s worth investing various solutions, if that was not the case I would have animated something by hand as you pointed out !

@Lane, Thanks !! I’ll see if I find this equivalent of FaceFX, maybe it will be less expensive !
This camera “sync” stuff look interesting too ! maybe not the best stuff for me as I prefer to work only with the sound, even if it’s less precise.
Indeed there are nice solutions for other apps, I’ve found one for after effect that look quite amazing given the input is only a sound, something similar for blender could be a great addition to the toolset.

If you have 60 bucks maybe f-clone http://f-clone.com could be something? I haven’t tried it

I’m not sure if this is what you are looking for. Also I do not know if it is compatible with the current version of blender:

Moved from “General Forums > Blender and CG Discussions” to “Support > Animation and Rigging”

There is already a Sound Bake routine in Blender, also a Sound Bake system in Animation Nodes and I have written a MIDI Sync addition to Animation Nodes, check out some of my recent threads in Animation and WIP sections here, or my Youtube channel and let me know if you want to try this approach. A lot depends on what you want to achieve…

Cheers, Clock.

Thanks a lot for your advices !

@bigbad : I think it was a kind of freeware first, that look interesting… For what I’m looking in it’s not appropriate, I want to use sound only…
Seems like a great tool anyway.

@Yafu : I think what you’re suggesting is similar to papagayo , so text based and you do the sound/text synchronisation by hand.

What I’m looking in is a way to do something similar into blender : https://aescripts.com/auto-lip-sync/
maybe a bit refined, to me the mouth is moving a bit fast, maybe something that average frames can be added.
I’ve also seen a script for afterEffect that output something much more convincing, but I haven’t bookmarked it.

I’ve tried with animation nodes and get a very simple thing working, maybe that can do the job, I don’t know much about sound manipulation so I’m a bit lost.

I’ve also found this page :
https://answers.unity.com/questions/139323/any-way-of-quotautomaticquot-lip-syncing.html
With a bit of theory and general information, but now I need to convert this into blender/python or animation nodes…

@clockmender : thanks , I’ll look into it !

It’s an interesting idea, and the Ae script looks pretty ok for what it does. I like the thought that you could set up frequency parameters to help recognise different vowel and consonant sounds. Does anything do that?

Hi,

I’m also looking into automatic lip sync options for Blender. Is there maybe also a way to automate lip sync (and maybe also facial expressions) using video footage? Maybe using Blender’s motion tracker?

Thanks.

hum, about auto lipsync I haven’t found something that meet my requirements so I build my own solution, it’s not perfect but it works for my project :

For facial mocap, there are attempts with blender’s motion tracker but it’s kind of a cheat :
some tests are good like this one : https://www.youtube.com/watch?v=jwJhwcPFh70
some are not so good : https://www.youtube.com/watch?v=a_AwDv9Al28

You can also use that : https://www.youtube.com/watch?v=FMGoXhPlncs
But you may need to write an importer for blender.

1 Like

Baking F-curves to mouth bone