Soft body not colliding

I am doing a basic physics simulation where I want to take a person, and do a slow mo of rubber bullets (for now, paintballs in the future) flying through the air, and causing ripples in their face, kinda like this

I set collision and soft body to the face, and collision to the “bullet” and have the bullet set to move towards the face via keyframes. When I bake the soft body, then render all that happens is the bullet moves forward and stops, while the face just kinda sits there and wobbles up and down.