As Richard Culver implies, your cloth object is actually working exactly as expected from the way you have things set up. It collides well with the body mesh (as long as it isn’t being animated – that challenge comes later) and looks very cloth-like.
But a shirt isn’t just a single piece of cloth, it has pieces and seams and structure that gives certain parts, like the collar, a different response than the rest. So you have to imitate that by using the parameters Richard mentioned to make certain parts act differently.
The buttons are part of your cloth mesh object but are separate geometry, so they drop because they have nothing to keep them attached – they’re responding to gravity. If you keep the buttons as part of the cloth object, you’ll have to find a method of “stitching” them onto the main shirt mesh by merging some common vertices. You can then use stiffness scaling and pinning weight adjustments to get them looking more natural.
But another way is to make the buttons totally separate objects and then using vertex parenting to “attach” them to the cloth. This takes some planning of the shirt mesh topology so you have vertices in the right place to do that, but it works well.
You currently have no vertex groups on the cloth mesh, so there’s nothing to modulate how the cloth sim and the shrinkwrap affect the mesh – you can create vertex groups and apply various vertex weights to control how the mesh responds to the two modifiers. You can also have the shirt move under armature control, which will be useful when animating – pure cloth sim rarely works well for this. But it’s a complex task balancing all these influences, so take Richard’s advice to heart and don’t expect it to happen in an afternoon.
