Shatter Part of Object (Problem)

Hi. I try to make a collision scene. A hitter object and a fracture collection. They both have required rigid properties and hitter (let’s call it a ball) ball has been keyframed. When the ball hit the group on one side, all the parts becomes active and fall down. But I want only the parts fall which are close the ball and others stop there.
I hope I could have make myself clear.
Thanks.