So for everyone struggling with a similar problem, I solved it and here’s my solution:
- Set up the cloth sim on the desired mesh.
- Create a vertex group for the areas that are attached to the solid parts and use that as a pinning group in the cloth sim.
- Create an armature with a bone for every solid part.
- Parent the mesh with the cloth sim to the armature with automatic weights.
- Make sure the bone of every solid part has a weight of 1 influence to the neighboring vertices of the cloth mesh (same as the pinning group).
- Make sure the vertices of the pinning group are only affected by the bone that controls the attached solid part. For example: Remove all influence of the left eye socket bone from the vertices of the cloth mesh around the right eye socket.
- Move the armature modifier BEFORE the cloth sim.
- Set up rigid body simulations for the solid parts.
- In pose mode, setup “child of” bone constraints to parent the bones to the right solid parts.
- Start simulation.
It might cause some stretching depending on the gravity strength on cloth and rigid sims but for this gave me the best results. A little issue I couldn’t fix is that the cloth doesn’t affect the rigid bodies so they just fall through. After some research this seems to just be not possible at the moment. There are some workaround with collision and soft bodies, but I didn’t want to kill my computer
For me the best way of fixing this was just shifting around vertices of the cloth mesh with proportional editing (after applying the armature and cloth).
If you want to read more about that, here’s a good source.