Getting rid of aliased edges in transparency (background).

I have an interior scene with windows and a background that is being rendered as transparent. The edges of the foreground/background are aliased. I want them antialiased though. I am using the progressive integrator. I thought the OpenGL AA setting is responsible for antialiased edges but apparently it is not, so which setting do I need to change in order to make it work?

Found the issue: My background was set to the standard diffuse color and had a strength of 0. Changing it to holdout fixed it.