You can absolutely manage this with cloth sims.
The problem is that cloth sims are notoriously finicky.
First, make sure you scale everything to about ten times “actual” size. That is, scale so that 1 meter == 10 Blender units, not one.
Next set up your collision collection. It should have ONLY those things that the cloth needs to collide with in it. It should be filled with PROXY objects. The proxy objects themselves should be simple representations of the object they represent, so, for example, the (person?)’s body should have a proxy object that more or less represents its shape and has a subdivision level close to the subdivision level of the cloth. (It would appear the you only need a person proxy and a staff proxy.)
The cloth should be composed of regular squares — not triangles, or ngons, or any other weird topology. It should be absolutely boringly composed of regular squares. The proxy objects need not be squares as well, but it helps if they are.
Set all the collision distances as low as they will go.
Don’t forget to enable self-collision. Again, the distances should be as low as they go.
Set cloth calculations (Quality Steps) to somewhere between 10 and 30.
Set collision calculations (Quality) to somewhere between 8 and 12 (no more, no less).
The cloth’s per-vertex weight should be somewhere between 0.01 and 0.10.
Avoid messing with the other parameters as much as you can. If you are experiencing excessive jank or jitter (or other weirdnesses), check out Edin Spiegel’s video about getting rid of it.
After that it is just playing with it to see what works best.
And, unfortunately, each different animation will require you to play with it again. You can make it perfect for one animation, but if you start a different animation then you’ll have to dink with things again. The good news is that once you get a good animation the first time, the parameters for most other animations will be similar, so getting the first time right is the hardest.
Good luck!
