Using VSE alphaover node to overlay models into movies

I have been compositing images & movies together using the “alpha over” in the Video Sequence Editor. The movies are the background and the images (which are PNG sequences) are blender renders.

I have noticed that even though the top images are RGBA output from Blender, making the “sky” transparent, it’s not really effective unless the sky colour of the original rendered images is black. If so, then I get what I expect; my blender model floating in front of my movie. If not, like if I use a grey colour background, then the alpha over mix has the background partially faded… more like it’s adding than going by alpha value.

Why is this? I thought an alphaover would go by the alpha chanel, regadless of the colour? Is there something wrong with my workflow?