Rigid body with cell fracture is not working properly

I am trying to make a scene wehre the wall of a room breaks up. I use cell fracture to brake the wall and i make them rigid body so they can be affected by the force field (with no gravity). everything works great upt until this point. But when i add some frames to the wall (it is not intersecting with the wall mesh) every thing explodes.
I have tried to set them all to convex hull, box, mesh, set the margin to 0, apply scale, but nothing works.
Anybody had a simmilar issues ?

Here is the blender file if intrested.

Blender file

Hello and welcome to BlenderArtists.

This does not happen for me when attempting to add keyframes. However, when the animation timeline is at a time other then at frame 1, then yes, everything does “explode.”

This is normal behavior as your simulation is synced with the animation timeline and it is playing through its physics simulation. If you wish to keyframe your cell fractures without them in motion, you will need to prevent your dynamic moving ball object from colliding with them, at least until you have keyframed your wall objects.

You could also remove all of your wall object’s physic types to relatively disable all of their physical interactions for the time being.