Hence “Othello,” which is what I’ve dubbed this rig (reason hopefully obvious), which is designed to emulate (but not simulate) the actions of the facial muscles. It’s intended to be principally for naturalistic expression & lip synch, but I’m sure could be adapted for more extreme caricaturization. I originally thought of building it for the purpose of manipulating the mesh (which is derived from my “Pugilist” model) with the intent of producing shape keys from the deforms, but it works pretty well on its own so far, at least in this rough proof of concept.
For this first test I’m using a low-level mesh density with Level 3 Subsurf – adding a multires layer and some more detail weighting will improve overall mesh response, which is a bit sparse in the forehead and nasal regions. The more dense mouth and eye areas show that already.
I used the excellent ARTnatomy site as reference for the facial muscle data.
Still thinking about a controls interface – haven’t done many of those since it seems an additional layer (or three) of complexity I don’t need for my own use, but I’m sure would be beneficial in a team-production environment.
The action constraint can easily be added into your setup. Since bones can be transformed in addition to the action constraint, a few preset mouth positions can be created using it but the controls will remain ‘freeform’ style as you have it. Thanks for posting a video of your setup
@chipmasque
Very nice rig! Thanks for showing it, and thanks for the artnatomy reference!
@all
In my opinon the only part of the face that is actually difficult to rig is the mouth. The upper face works perfectly fine with just bones and shapekeys. It is the mouth that is hard. And the reason is the sheer flexibility of it. Because of it’s rubbery/stretchy nature (yes, even in reality ;)), gliding along the surface of the teeth - but not limited to the surface of the teeth, it’s motion is very difficult to define and breakdown.
Especially since the mouth influences areas upwards on the face (try pursing your lips together completely to one side and watch your nose rotate 45 degrees!). Because of this falloff-like effect on the nose and cheeks, I think a deforming bone/meshdeform base for the muzzle area is essential.
This is also my problem with shapekeys - there is no falloff! The shapes are isolated entities, and yes with great skill they can be blended together to create the illusion of a falloff. But all too often it doesn’t work out that way and it starts to look both uncanny and stiff.
In the image above he is saying “allllrriiighty then!”, though it is a little hard to tell without a tongue. Very little has changed about my setup from what I posted before. There are some key differences I will list here:
segmented B-Bones for lips removed, previous lip controls are now the deformers
inserted an extra edgeloop on the meshdeform cage in the corner of the mouth
No shapekeys.
That is correct, there are absolutely no shapekeys on this rig. The meshdeformer is controlling the entire muzzle area. I think some corrective shapekeys could enhance it, but I’m trying to get as far as I can without them. There is still a lot of work to be done on this before I post a .blend, but I just wanted to show you guys that there is hope beyond shapekeys
I dont believe that you not using one shapekey in that animation my friend, is just awesome…
My big poroblem is always the “U, B, C” phonemes, when you going to release a video tutorial or maybe a pdf article…?
Great deformations of the orbicularis oris regions, feelgood. One thing I learned when building the Othello rig (besides many muscle names!) is that the upper lip motions are connected to the lateral and upper nasal regions (flanking the bridge of the nose) by a couple of long flat muscle, causing a linkage in the motions that you could incorporate to keep the expressions more nuanced.
Can someone explain to me the drive to eliminate shape keys from this process? I understand it from the perspective of a technical challenge, but what does it win you from a production stand point?
Not so sure it’s a matter of eliminating them as much as supplementing or augmenting their use, or placing them in a less dominant role in the production of expressive deformations.
As I mentioned my original intent for the Othello rig was to provide a means of generating shape keys based on muscle deformations, as opposed to directly sculpting them all. In that regard it is (imo) fast and reduces the amount of sculpting for any one shape, plus having a grounding in physical anatomy.
But it also works well on its own, though it isn’t yet a complete solution – I’m sure shape keys will have a place in its use as well for non-muscular deformations and for tweaking nuances and generating very small details like skin folding and wrinkles.
My own preference for a bone-based approach is also because shape keys, to be time-effective in production, require a control interface system for the animator, something that requires substantial time to build, test and troubleshoot. Using the Action window sliders and/or IPO curves is not terribly intuitive. Whereas selecting a bone and setting a Transform value or adjusting the manipulator right there on the model is a very direct and intuitive way of adjusting a deformation, not as speedy perhaps as a well-designed and executed controls interface, but also not requiring the development of such a system.
Can’t see that development of alternatives to shape keys is in any way counter-productive, just adds tools to the toolbox.
Exactly :yes: I think we’ve already established clearly what shapekeys are capable of, and also what their limits are. I’m far more interested in looking into alternatives. If you always do what you’ve always done, you’ll always get what you’ve always got. Time to innovate
Thanks, that is very helpful! The more ideas on organizing the mouth motions the better. The biggest challenge right now is trying to design the mechanical setup and control style. Right now what I have is very freeform, and I’m trying to define the motions better. For example, since the lips are wrapped around the teeth a large part of their motion is created gliding over the teeth surface in an ovoid semi-circle. Not sure how best to do that (if at all), but these are definitely the thoughts running through my mind
Well I have to build it first, haha! ideally more than once too a month or two at best… I have a series I’m working on right now, and the face will be the last thing I cover…
I guess usefulness is in the hands of the user – I’ve done a great deal of work with shape keys in the production of the final cloth effects for Kata, and find them extremely time-consuming to edit and manage. So far, by comparison, my work with the Othello rig has been very fast, and gives me excellent and immediate control over the facial deforms, with easy tailoring of all general shapings for any one passage in a sequence – essential for finely nuanced emotional expression. I’ll post the new rendered vid soon. Far as I’m concerned, if it works, it’s useful, though it may not be everyone’s cuppa :D.
Timothy Albee has done some development of a plugin for LightWave that uses shape keys. I was going to use it until I dropped LightWave from my project.
However looking into how he set it up may give some inspiration.
Pooby’s rig ideas do have a tendency to get over complicated. However he is a genius at rigging and I’ll give him that. And his animations look exceptionally real.
He was using LightWave and dropped it for XSI when the LW CA tools were too limiting for where he was going.
I think the control plugin that TA developed looks very promising. It is apparently very quick and intuitive as you can animate in real time.
Yeah, it’s a start, but it’s not really a muscle sim since there’s no allowance for the way muscles change shape in cross-section – thickening and the like – as they contract. That’s a very good place to incorporate shape keys, unless you want to do a full-out muscle sim (!!). But using bones to approximate muscle action (by scaling) and building opposing (antagonist) muscle pairs seems to be a viable way to go.
By no means would I consider “doing away” with shape keys, mesh deformers, or any other tool, but I have a preference for using bone rigs, so that’s where I started when looking at the idea of facial rigging. Looking at a possible workflow for the Othello rig, I can see using it to set up the majority of the facial mass deformations for a sequence, then translating that to shape keys for final sculpt-level adjustments and detailing. That’s the path I’ve been using for the Kata cloth effects and it gives a lot of control over the results. For some applications it could likely be used without the shape key step. I need to apply it to broader uses than just this one test sequence to see how it holds up, though.
IMO, there are few things that I have not really liked about shapekeys:
Flexibility for forming expressions - so far, the shapekeys I’ve mostly seen in motion have resulted in severely limited and wooden limits to the expressions able to be formed. I suspect that this is not that obvious with “big studio” work a lot of the time, since the animators are just made to blend several corrective shapes together manually on top of a base shapekey which might get 40% there, requiring another to push to 70%, and so on. To be able to achieve enough range in the extents of the shapekeys, also taking into account non-linearity, probably would require more than 1 shapekey+control.
In contrast, with bones, IMO, you can place/rotate/scale some main ones to get about 90% of the way there in one big swoop. Then, probably with fine adjustment bones (or maybe some shapekeys), finish off the job.
Topology dependence - probably this isn’t so much of an issue on a linear production workflow, where models do not get their topology changed after a certain point, and shapekey+UV work can begin, but having to regenerate large masses of shapekeys is a bit of a turn off. Probably could in future be alleviated by better use of various projection + transfer tools and/or different workflow styles…
The large number of them required - probably this isn’t really that bad in comparison with the number of bones in more advanced setups with multiple layers of controls, and might not be that bad when compared with fighting weigh painting issues. However, in the end, you still need to create quite a few to be able to achieve the level of flexibility required (see above).
Probably a really personal “old fart” regressive excuse, but back when I was first experimenting with rigging in Blender, shapekeys (more more accurately, “RVK’s” and their “absolute” cousins) were totally impossible to understand, especially in contrast with bones+vertex groups.
They sometimes worked, but only up to a point, and sometimes would just get corrupted in some way… and don’t get me started ranting about shapekeys being represented as horizontal blue and orange lines spread out vertically in a graph editor… ick! This was also the era when driven keys had not yet been added, and editing shapekeys was manual vertex pushing on a 1 by 1 basis (usually) to get things in the right places.
Another excuse in a way (from developer standpoint, partially) is that shapekeys have always seemed to me quite a hodge-podge of inconsistencies, instabilities, and limitations. As a developer, I should probably address that ;), but as a user, umm… give me my bones bag any day!
Control interfaces - bones = simple visual controls ON THE FACE. Shapekeys = slider panels either off the face or somewhere else in the interface. We could consider custom modal tools to tweak various shapekeys at once in the 3D view per control now, but the effectiveness of this we’d still have to investigate
I’ve probably forgotten to add a few of the other things I didn’t like about shapekeys here, but this is a good sample of quite a few of them. Probably a lot of these issues aren’t as much of an issue now, and perhaps won’t be in the future either, but they are still issues that come up everytime I think or hear the mention of shapekeys.
Don’tcha luv it when you’re old enough to say this, and get away with it!?
I have to say I agree with most of your reasons to a large extent. Though shape keys are extremely useful, and essential in some situations, they aren’t by default the way I’d choose for doing facial deform animations.
The rendering of the complete “Othello” sequence finished up this morn’ – lots better lip sync and expressions, plus some body language (limited due to the nature of the model – he ends just below the frame line!), but still not 100% satisfactory – I need to use this full test to analyze how the weighting is working out and to plan where to use some corrective shape keys – mainly in the area of the naso-labial folds where the expressions should stretch them out flatter, hard to do with just weighting but a perfect place for some shape key application, I think.
Currently doing a UV breakout and some costume modeling, plan on some further sculpted refinements to the model detail as well using a higher multires level. That’ll show up in another thread, though, where I’ll also make this test-case .blend available for d/l for interested folks to play 'round with.
It might just be the way I animate, but ‘on the face’ controls have always proven to be a real pain to actually animate with. The controls themselves get in the way and obscure the face.
The kind of mesh highlighting setup cesio wrote, inspired by the method they’re using at the studios in Hollywood, looks absolutely ideal (wink wink wink). Short of that, a 2D control panel with joysticks like in ‘Stop Staring,’ probably best attached to the viewport camera, is much clearer and easier to work with than floating ‘on the face’ transform controls like the first Sintel face rig.
The idea that shapekeys lend to a mechanical performance as opposed to bones seems backward to me. I think the lack of ‘morphing’ caused by bones results in a mechanical performance, and getting mechanical movement with a good set of shapekeys (like 12 or so shapes with asymmetric controls for a human face) is undoubtedly just a poor job on the animator’s part. This is especially easy to do with animators using the controls like on-off switches, instead of using the whole range between 0 and 1.
I think I hit the wall with bone-based face rigging, like in Cessen’s spot-on explanation at the top of the thread, and had to take a break. Haven’t built a shapekey setup yet, but I expect that in 2.5 it’ll be relatively painless compared to 2.4. Right now I’m itching to build control interfaces for the Sintel model (thanks for the blend!) but haven’t gotten drivers figured out yet.
I remember someone had a demo blend where a bunch of small floating bones rode on the surface of a face deformed by shapekeys, using the first Sintel model. This seemed really interesting; did anything come of it at Durian?
@chipmasque
Wow! That is a really good bone rig! That is without shapekeys added? I for one would be very interested in that .blend :eyebrowlift:
@bunny
I’d be very interested to see what you could do with a shapekey based rig. I was following your experimentation with action constraints, sorry to hear it didn’t work out for you. I thought it was looking pretty good, what went wrong?
In terms of why find alternatives for shapekey-dominant methods, this playblast of the next step in the project illustrates one good answer – non-dependence on topology:
I put a level of multires on the original model and sculpted him up into a much less stylized character, but haven’t done a thing to the facial animation, or even the facial weighting. The increased cage rez makes a lot of the expressions more pronounced (some parts even too much, as in the corrugator muscle area above the bridge of the nose), and there needs to be some weight polishing (had to do that with the body rig already), but imo that’s not nearly what revising a full set of shape keys would require. This seems a useful capability for a production team, as the animation could proceed using a lower-rez proxy (but still a good representation of the character), while the final characterization sculpt is still in progress.
