You need to save as OpenEXR or composite it within Blender. It’s the same issue being run into here: Rendering an ACTUALLY transparent Glass? (Cycles)
Straight alpha isn’t able to do the sort of composite you want, only premultiplied alpha can.
As for speed rendering the whole thing, if you are using Blender 2.83 or the 2.90 alpha, you can enable adaptive sampling in the render settings. This will reduce sampling of empty areas to a minimum (some samples need to be done still to even know what is an empty area and what isn’t)