I imagine you’re using scribble control net for that?
I’ve found that each control net (and each version of that control net, for that matter) has its own quirks and talents. They’re each like an apprentice that does some things well and other things poorly. And they’re each as useful for how much they leave unspecified as they are for how much they specify.
I’ve most recently been working on a blender file to inform composition. Depth is the most effective controlnet I’ve used. It’s good for characters and for non-characters (I once handed it a cloth displacement texture and was surprised to see that the controlnet agreed that we should be drawing cloth.) Preprocessing is slow, and it’s not precise. But that’s kind of a good thing: I can create the depth map in Blender in an instant (an Eevee depth render), and I can run the controlnet at a high strength, until late in the process (like 0.95 strength, running from steps 0.0-0.9), and so the depth can take strong control of the composition, while still leaving room for SD to draw details that depth doesn’t have the precision to cover.
To that, I’m adding a custom openpose rig (the existing rigs aren’t reliable; I’ve added occlusion and am projecting onto the near clip plane for size control), and a base render for mediapipe face work (which isn’t reliable enough, unfortunately, but is still better than openpose or nothing) and an early (steps 0.0-0.2) tile to get base color, without having to worry about the details of that coloration.
With this, I can get unusual poses, like doing the splits. There are still limits. Post-process cropping is still smart. Reinforcing the controlnets with prompting is still wise.
But I end up with something that looks a bit like this:
Obviously, I don’t expect that pic to make sense, with a mess of nodes and noodles and cropped-- just showing the kind of inputs I’m using and the outputs I’m getting from that.
Notice that the depth output gives me reasonable control of the hair, the sword, the earrings, without demanding that I create the muscle definition in the Blender file. I’m intentionally giving it a vague, weak background depth map to let SD dream. The Blender file’s all set up with compositing nodes and view layers to give me the exact images I want with a single press of f12, in less than a second.
No hand fix, no face fix here. I’m rendering at a high base resolution, which is fine, because the controlnets fix the composition. The hands aren’t always perfect but the combination of controlnets means that they’re right more often than not.
Depth and tile are strong controlnets that can easily overdefine the image. Pose is a weak controlnet. Lineart and canny are other strong controlnets. Scribble has felt like a weak controlnet to me.
The alternative to this is, generate 16 times and pick the best one; then inpaint everything that’s wrong with it. And each inpaint is probably 16 generations, picking the best.