Masking layers same speed as not masked?

I masked a few layers out of my final render to speed it up - I thought it would only render the opaque parts, but from the speed it seems like it is trying to render whatever is behind the alpha channel.

Has anyone else had this? Any way around it?