as you say, shapekeys can go way beyond just additive linear offsets. its a tall ask, but something along the lines of the weta/gollum combination sculpting system they created would be an ideal end goal. you want a system that lets you sculpt and correct combinations of shapekeys, store rotational offsets, proper spline interpolation as well as linear, and wrap it up in an intuitive interface so that modellers rather than just riggers can easily get into it.
the weta system was a quite complex network of nodes and melscripts, someone else has done their own take on it as a single maya deformer called BCS. The videos looks amazing:
Joe Alter has his own system that focuses more on the puppeteering/performance side, but it does feature spline interpolation between shapes, making for much nicer deformation:
A possible workflow for all this within blender might be:
- select your mesh, add a shapekey modifier
- create a new shapekey called ‘smile’
- blender duplicates your mesh, renames it smile, adds a ‘smile’ slider to the shapekeys UI.
- setting the smile slider to 100%, and modifying the smile mesh will be updated live on your main mesh
- switch to the sculpting tool, sculpt in a nice smile.
- when done, you can either delete the smile mesh, and shapekeys will store the offsets internally, or just hide it, so you can continue to edit it ‘live’ later. if you delete the shape, an ‘edit’ button next to the smile slider would recreate the smile mesh for you to edit again.
- set the smile slider to 0%, create a new shapekey, ‘jawOpen’
- blender duplicates the input mesh, names it jaw open, makes a jawOpen slider
- set the jawOpen slider to 100%, sculpt this as before
- within the jawOpen slider would be an interpolation drop-down, set this to ‘rotation’
11. turn both ‘smile’ and ‘jawOpen’ to 100%, get a horrible additive mess, decide to fix
12. click the ‘fix combination’ button. this could either duplicate the mesh in its current state like before, or just work in-place whatever people prefer I guess.
13. sculpt away with the sculpt tool
14. when done, shapekeys knows that whenever the current configuration of sliders is reached, to display this particular corrected sculpt. this correction shouldn’t become a 3rd slider, it shouldn’t be scripted with python, the user shouldn’t have to manage correction states, it should all just work.
for advanced users they could open a ‘combinations’ subpanel, that’d be where that ‘smile+jawOpen’ shape would live. it’d give info on what primary shapekeys are connected to this combination, control over how it should start blending and with what interoplation, toggles to enable/disable correction.
i’m sure there’s things I’m missing, but that’s the jist of it. other things that’d be useful would be setting the smile slider to 50%, right clicking and choosing ‘add shapekey’ so that you could have a more nuanced smile animation; the cheek puffing out before the smile reaches its full extent, that sort of thing. these would be represented as ‘ticks’ within the slider as if they’re keyframes, allowing you to set their interpolation type.
and that’s my two cents.