Are you sure you need a displacement ? Maybe a bump is enough…
Anyway: I think you displaced not only the side of the brick (any axis) but along normals… if you really want this the edges of you holes maybe should have some bevels… because the displacements “jumps” over the edges…
The displacement is to enhance the texture of the brick. It looks all flat when there’s no displacement on it. But I think it’ll be fine with a very subtle displacement and camera blur.
It was more of a question if there’s a way to use fine gravel displacement on objects that’s not just a cube but for example with holes or other booleans