AFAIK avi-jpeg does not have an alpha channel.
If that is (apparently) true, than the RGBA button should really be disabled if avi-jpeg is chosen (or jpg/single image which also doesn’t have an alpha channel).
Apparently some avi-codecs do have an alpha channel, I’m testing the ones on my system now, but don’t have any definitive results.
Do a Google for ‘avi alpha’ or ‘alpha channel avi’.
The easiest solution is to render to .tga or .png then assemble the images with the sequence editor or an external program like Virtual Dub.