How can I make sure the scene is empty? I mean, will deleting the cube do that, or do I have to remove other objects, like lights? Vegas has the F-curves built right into each track in the sequencer. It’s really nice…for a trial. 
Well you get what you pay for…
I think that you can delete everything (including the camera) from the rendered scene.
@3point: I’ve just tested it (latest build from Fish) with the VSE and it doens’t render anything, it simply produces a pcm file if you hide the video strip. If there is no image input there’s no rendering involved and it goes lightning fast.
I also don’t see how the discussion about scenes is even relevant in this context. I thought the OP referred to extracting the audio from a video in the context of the VSE. The proposed solution by yellow plus hiding the video strip(s) works fine on my system. Can you confirm this?
Okay, so you actually have to add an image to it and then hide it for the audio only mixdown. THEN you have to pick H264 codec, and THEN you have to pick an audio only container and THEN you have to pick a codec. AC3 is apparently a codec that can be put in another container, but I’m pretty sure I’ve been encoding stuff with AC3 only sound, with no container. Blender crashes during the encodes occasionally.
@Joel: this thread is getting hard to follow. I don’t see any correlation between the thread title and the latest posts. What is it exactly that you want to do? Are you working in the 3D viewport or in the VSE?
Mixdown produces an uncompressed file. Haven’t tested it with any codecs so I cannot say if it works. My experience is that rendering is prone to fail when trying unconventional container format/codec combos which might be what happened in the case you’re describing.
@blendercomp it used to be that Blender would render the scene frame (3D) first then add the strip frame to the output. If that has changed then great!
BTW this sounds very script worthy, unless its going to be a core function that Ton doesn’t want as a script.
@blendercomp: I’m only trying to use the VSE. The only reason I’m going into the 3D viewport is to delete the default camera, cube, and lamp, because then it tries to render it instead of just rendering out the audio. @3pointEdit: No, it’s still crap. It still renders the scene frame if you don’t remove the default 3D elements first. Just tested this out with 2.58. Almost a year later and the software is just as user hostile as it was three years ago.