Holdout shader takes too much rendering time

Blender v2.79, Cycles engine.

For compositing purpose I have many layers with relatively small objects with big masked areas, which take half of rendering time.

For example with high samplecount (500-1000) blank/empty screen renders in ±20 seconds and just 4 sec with 1 sample.

What can I do to speedup it without dropping samples?