Not too much information is normally shared on Shape Keys - specifically how they are used in game engines (as Blendshapes and Morph Targets). And it should be common knowledge that using too many shape keys in game engines causes extreme performance loss when certain thresholds are crossed.
I’ve been working on a method of shape key reduction using Principal Component Analysis PCA via Singular Value Decomposition - which in theory can reduce say 52 shape keys into ~16 with minimal quality loss <1%.
I’ve seen this mentioned in SIGGRAPH papers and talks for AAA games like Uncharted and The Last of Us - but I’m curious why this is not more commonly utilized in the indie community?
Metahuman does something similar - but seems to be a black box, and I have not found any open source tooling to perform PCA on shape keys (or in general).
I’ll be updating here with the research and demos once I have this verified to work (specifically inside blender and with Unity as my initial target)- and intend to release this as an extension in the future.
Curious on thoughts on this - it kind of irked me when I saw vtuber rigs for ARkit use 52 shape keys (absolutely ridiculous to me, but it seems to be extremely poorly documented in unreal or unity documentation that this causes such a performance impact - and on top of this, the internals for GPU skinning in unity and morph targets is also underdocumented – it will kick to CPU at unknown limits based on your mesh, your scene, your specific scenario – somewhere around 8 concurrent shapes on a mesh with ~20k verts as far as I can tell)
It’s obviously an issue major AAA studios have solved, mention, and promptly black-box, but isn’t that crazy, just a little linear algebra.
It’s really not as dire as you say. We have 5-6 characters with 52+ shape keys each per Unity scene at work and we have no performance impact whatsoever. That’s not to say that I’m not interested in what you’re doing, it sounds very cool, but 52 or more shape keys is pretty much the standard for game development at this point
Curious what’s your target platform? The CPU hit is harsh but on gaming PCs and consoles it is OK-ish (not batchable and will not be GPU even if using GPU skinning with those kinds of numbers- incredibly hard to debug)
You won’t see impact immediately (without any active/animated blendshapes) but if you have say 20 concurrently animating on 5 different characters, or port to switch… Or mobile… you will see what I mean in your skinned mesh renderer cost during runtime when utilized. This is exactly why I bring this up, it’s a hidden bottleneck that seems to be too often brushed aside. It’s not seen until you are at scale.
This model got ~120 shape keys for the face, however they all got their own functions, meaning the rig controler’s are going to drive each of those shape precisely :
for instance this controller scale X allows to scale the lips using two shapes ( scale in / scale out)…
Those 2 cross bones location drives 4 shapes each, (8 for the two) …
+X-X allows the lips to extend or compress, while up/down drives a smile/frown shapes… And I’m not 100% sure but there are probably intermediate shapes to smooth out bad deformations…
Anyway, if each shape got a particular function tied to the rig, how can you manage to have less of it ?
Maybe this works only for realistic faces driven by mocap , and you can simplify it by having more general deformation and remove fine grained ones. However I’m wondering what append to the rig then…
At least I suspect ARP allows also to rig cartoony characters where you want local control over every part of the face…
Yep it is tricky. My initial focus is on the 52 ARKit standard face shapes to get this working as a POC - but you bring up a good point- how do you control as an artist the reduced PCA keys?
This is done via a projection coefficient matrix. This is essentially a list of all your original shape keys and the percentage of each of the PCA reduced shape keys to reconstruct each of your original shapes. This will make more sense when I upload the support (likely later this afternoon).
Initially I will create a separate shape key menu which will look identical to your original list of keys (all 52) to animate / use. These will drive the PCA reduced set based on the matrix. Further down the line, I’ll have to integrate some kind of keyframe remapping to drive the PCA, for if you have already animated (via bones or directly) and remap drivers to control my custom menu instead of direct shape keys.
I was kind of tongue in cheek saying “just linear algebra” - it is a very complex project to get right. I’d honestly be really interested in testing it out on a file like your example once my baseline is tested and working, I don’t have a big bank of “real world” examples I can pull from right now.
Ok that makes sense then !
we’ll still have original reference to the shape keys and and maybe for 1 of the original shape you’ll call 3different PCA shapes maybe with positive and negative values to form the original back ? something along those lines ?
Pretty cool ! I’m curious about what this would mean in terms of performances in blender ! Since it’s always good to have realtime feedback when animating.
For test files feel free to poke me and I’ll see what I can do. Also blender studio is currently working (if that’s not already done) on a shape key based character rig so it’s probably worth checking that too if you need good examples !
Also your 100+ may very well not compress cleanly down into 16. That 16 number is a generalization, the tool currently let’s you choose this number. So stylized characters maybe we say 100+ compressed to ~40? There’s some inherent quality loss if each shape is in fact completely unique. The 52ARKit might even compress cleanly into less than 16 for example.
This is the exact idea, it references a CSV that’s says smile-left or whatever your original key is then in columns it holds negative and positive values for each compressed key to reconstruct the original
Ok !! cool ! very interesting , i’m super curious about what kind of results this would gives ! I’m also wondering what it would represent in terms of drivers / depsgraph evaluations. Since in one hand we remove shape keys but on the other we create complexity to form them back and keep them linked to the face…
Or maybe it’s a non issue… probably there are some cases where it would work better than others…
I think on a realistic face everything is so much organic and shapes are far less extreme that it might be easier to find redundancies across all shapes.
In a cartoony rig everything is far more separated , but maybe the algorithm will manage to figure something out !
I totally understand that the results will be variable and it’s not possible to know in advance how much shapes we could spare… I suppose there will be some kind of a tolerance factor like when compressing images so you can find a good balance ?
Yep that PCA Basis value I’ve been referencing is essentially the compression quality (how many shapes should it try to compress into)
And what you are asking is exactly where I’m at right now, even to me, a linear blend between 2 PCA shapes describing specific shape keys seems like there might be weird artifacting or deformation (but in everything I’ve read so far this is a non-issue) so I’m excited to figure this out and report back.
In terms of depsgraph… It could be heavy- because it’s an additional step, less overall shapes but still an additional compute step to figure them out. My focus is for game engines though. The ‘correct’ implementation of this long term would be moreso on export- you really shouldn’t have to think about it (like how metahuman just does something similar and in the background but user does not notice). Like on export convert all your shape keys, redefine your animation curves to PCA- in engine you’d see idk 8-20 shapes (whatever you define) instead of 100+ but your workflow for animation in theory should not change (in a perfect world)
Hum !! that makes sense ! Indeed the goal in video games is to have baked animation so you can simplify stuff here there too !
Anyway, curious about if this could be useful in an animation workflow, but indeed that makes the pipeline a bit more complex too as we would need to generate the PCA when publishing the rig, so we still have access to original shapes for editing and animator can use the PCA … Obviously this might be much more useful to a game workflow where it’s basically optimization for free. Where in a animation / VFX pipeline there might be simpler way to optimize stuff and it might not be worth it if it’s just about saving a few fps here and there.
Animators can always hide other characters for instance to save performances , which is impossible in a game where every possible optim is useful …
Anyway, interesting subject !! thanks for taking the time to explain all this ! I’ll subscribe to the thread to follow your progress and feel free to ask me if you need some test models !
ABSOLUTELY could cause turmoil in the pipeline. New shape keys and altering existing will require new PCA to be ran and all animation could very well break. Not much use in film because of this (I did see some comments of feature films using this in a different way, but I think that is more of an early 2000s mindset when PCs would stall with 200+ shape keys on advanced rigs). But yeah everything you are saying is on point I’m glad people can follow and I’m not just going insane.
Ok update: 52 ARKit shapes compressed to 16 worked pretty well! I’ll post images or video later. But @sozap was right in saying this will be difficult for extreme stylization. The most telling indication of this is “eye_blink_left” (and right) essentially do not come through whatsoever. This is because they are a small amount of localized verts that are not utilized in many of the shapes.
My plan is to first run at least 3 PCA tests:
52 → 16 (the one I’ve done)
52 → 24 (see if it catches blinking)
52 → 8 (see how far we can go and what shapes get compressed too far)
I think the final form of this may be a selection menu for shapes to keep vs shapes to compress OR I can use the PCA result to analyze which shapes did not get good representation.
Will be doing a little research into what kind of layering AAA does for this method (do they compress left side of face and right side separately? Are there shapes they always keep uncompressed like blinks and jaw movement? What shapes are typically bone driven rather than shapes when PCA is utilized?)
I finished a scrappy menu which controlls the PCA basis shapes by altering slider values mocking the original shapes, but needs some improvement so I can run ROM animation tests and comparisons without as much manual work. Thanks for sticking with me if you’re reading this, some cool images and video to support this research coming very soon.
This got me some good information (video on google drive going though all current compressed shapes vs original https://drive.google.com/drive/folders/1ZT4C08ZB2g3Y_ZNsBqYj7wmz29xUYKdz?usp=sharing). I will be running PCA on grouped sections and working on my algorithm - right now, it takes into account all vertices, where it should only take into account localized verts modified by a ‘specific set’ of keys (i.e. something like eyes, mouth, main). Each of these sets can probably be reduced to less, so I can get a result of 16 (my general target).
I still think this is impressive (minus the eyes) seeing as the face on the right only has 16 shape keys and probably 30 of these are passable even though my math is not locked in.
Hahaha ! That’s pretty cool ! Well done !!
Could be interesting to build a “difference map”, to see where there could be any issue. Anyway at first glance even if the shapes are a bit different it looks good nonetheless !
I just finished a UI to be able to separate out groups based on naming and apply specific basis numbers per group, on top of that, it will need to retain certain keys in full. Takes about 20 min each time I run the algorithm so I’m just trying to carefully chip away at this on a dedicated machine. Btw term for what your saying would be ‘heatmap’ and yes totally want to do this once I am closer and calculate visually and with a value the percentage error
The work continues. I see why no one has tackled this. Grouping keywords work wonders, but L/R splits on geometry without perfect mirroring still takes a hit, I’ve localized subgroups of groups to a act on left and right basis within groups.
