So there are a few things that definitely aren’t an issue. Any transformation, of any bone, means that the entire mesh needs to be recalculated, which is what causes the slowdown, and there’s no way around that beyond carefully dividing your armature and mesh into individual parts.
The thing that might be an issue is that any animation on the armature means that the armature is recomputed at every single frame, meaning that the mesh is recalculated on every single frame. This is something that could be addressed, but generally speaking, is pretty low priority-- generally, if you’re armature deforming something during playback, you’re armature deforming it throughout playback (even a very still character should still be breathing.)
The bigger problem here is Blender’s armature deformation, which has very poor performance, as demonstrated by this file; I’m getting 6fps on 180k armature deformed verts, no other modifiers. That’s pretty terrible when comparing to any number of different game engines, which are going to be armature deforming all on-screen characters (probably not 180k verts, but possibly close) at 30-60fps while running other code and rendering something much nicer than a solid view (including many other, non-deforming verts.)
So while I think Blender’s armature modifier drastically needs some love, the fact that the mesh is recalculated for any bone transformation when that’s not strictly necessary doesn’t seem like a particularly pressing part of the issue.