I kind of fixed it by deleting that cursed plane and extruding the lower border of the middle tower. I’m quite happy with it now, but I’d still like to know if there’s any fix for this in case I’ll bump into it again.
i had a similar problem
and it turned out to be caused by a Material i’d assigned to the object.
I deleted the material and the problem went away, AO baked just fine
dunno what was up with that.