Best Way to Turn a Structure Into Bricks/Pieces?

Hey all, I’m very new to Blender and I’ve been looking into physics simulations.

I’ve created several structure crumble animations using the cell fracture addon and using rigid bodies, which works great. However, this is done by turning the entire structure into split up and random pieces that crumble, regardless of whether another object hits it. I was wondering if anyone knows of either an addon or a way to turn my structure into a series of bricks/building blocks that will support itself while I have other objects hit it (e.g. a sphere slamming into the structure). Any and all help and suggestions is greatly appreciated!