Feelgoodcomics suggested that I start a thread here about the facial rigging issues I ran into on Durian. And I think that's a good idea. So here I am. :-)
As sort of a preface, let me first say that unless you plan on doing some kind of tissue simulation, shape keys are going to have to be a part of your face rig. Bone deformations just aren't enough to create the creases, bulges, and wrinkles of face deformations (unless you use thousands of tiny bones... at which point you're almost just doing shape keys but with bones anyway).
Recognizing this is really important because it influences how you design the bone parts of the rig: you need to design the bone deformations in such a way that shape keys will work nicely on top of them.
Shape keys get applied *before* bone deformations, which effectively means that shapes are applied in the local space of the bones deforming it. For example, if you make a shape key that moves some vertices left, and then you deform those with a bone that rotates 90 degrees, then the shape key will move the vertices down. Normally this is a good thing. If you rotate the entire head of your character, of course you want a smile shape to rotate with it. But if you have multiple bones on a face all rotating in different ways, this can severly mess up shape keys. That's not to say that rotating bones is completely off-limits in face rigging. But you have to be mindful of the "deformation space" it creates for the shape keys.
This was something that didn't occur to me at all when I first started rigging Sintel's face. In particular, I used a ring of stretch-to bones around the mouth to manage lip deformations (similar to my setup for big buck bunny). Unfortunately, this totally screwed up the shape keys when the mouth was open, because the lips (especially near the mouth corners) were rotating so extremely, creating a highly distorted deformation space for the shape keys.
And that's probably the biggest lesson I learned: design the bones to create a good deformation space for the shape keys.
And this doesn't mean "don't rotate bones". In fact, in some cases, rotating bones seems to create a *better* deformation space for shape keys. For example, a properly weighted jaw bone. And the same goes for scaling and translating bones: you have to think about how it will affect the deformation space for shape keys. Sometimes scaling a bone might be good, sometimes it might be bad. It depends.
But design the deformation space with that in mind.
At this point I don't yet have any really good answers for what a good deformation space is for a face or how to construct it, because I haven't had the chance to play around with different setups (which would take time, which is why we abandoned it for Durian). And I'm also pretty sure it depends on some of the larger design decisions of the face rig: if shape keys are just used for small details and corrections, then the ideal deformation space may be quite different than if shape keys are used for larger deformations.
But it would be really cool to explore that (among other things) in this thread.
Something else I would like to explore in this thread (which I have very little experience or inspiration about) is control schemes for faces. This is also something that is influenced by the nature of the particular face rig. I suspect the ideal control scheme for a very realistic face may be a bit different from the ideal scheme for a very cartoony face.
In many ways, I feel like the control scheme should be the starting point--not the ending point--of designing a face rig. It's how I approach most other rigging: figure out how I want to interact with it, then figure out how to accomplish that. But I'm lacking good ideas on that at the moment, so any brainstorming would be appreciated, and any examples we can find and share would be awesome.