Path Traced and Binaural Audio for Blender

Hi,
this is a quick poll on how interested the Blender community is in immersive 3D Audio.
If you want to see this stuff implemented at some point in the future, just leave a comment and talk to other people so we can get the dev’s attention.
Thanks!

Hey!”, you might say: “Blender uses OpenAL and offers surround sound already.
That’s true, but it’s not the end of the road - by far.

There are two concepts, which could make Blender a one stop option for creating animations with great spatial sound:

1. Path Traced Audio
Just as Cycles simulates light following an inverse path from the camera to a light source, sound can be rendered dependent on the source’s location, the position of the listener and most importantly: the environment.
Different materials reflect and absorb it, it diffracts frequency dependent when hitting an obstacle…
To make a long story short: Every sound changes before ever reaching our ears in the real world and even though it’s a big task to implement all these things in software, faking it is quite common in Game Engines.

Introducing the big boy variant: NVidia VR Works Audio

It’s GPU accelerated and even if it’s not integrable into Blender, it still shows how great path traced Audio sounds. I don’t know about you, but… I want something like that!

And it get’s even better!

2. Binaural Audio
As humans we are naturally bound to our own bodies and in regard to our listening capabilities to our ears - how they are shaped in particular. Thanks to them, we can precisely locate sound from all directions. Turns out our brain does this by simply getting used to the way certain frequencies are dampened when sound reaches our ears from behind, the front, the top or from underneath.
The catch is: One can “simply” make tons of measurements using a dummy head, record how the sound changes from every possible direction (see “HRTF” on Wikipedia) and apply this as a filter before playing sounds through our headphones.
The result: Even though it’s still just headphones, the sound is literally altered as if you are stuck in the head of someone else, our dummy in this case, and since most of us are shaped roughly the same, you will suddenly perceive it as very directional.

You may already know this, but it’s always fun listening to: Binaural Audiobook on Youtube

It’s not an easy thing to get the measurements, as a matter of fact, there’s not even a standardized file format for HRIRs as far as I know, but applying them within Blender should be rather easy on the other hand.

If you’d like to play around with existing software on the topic of binaural audio, check out “Audio3D” or the “SoundScape Renderer”.

8 Likes

I doubt the Devs will have time for it. Not in the near future.

Still… getting this kind of audio from Blender would certainly speed up things. Could be a gamechanger for animation workflows. Love the idea!

1 Like

I have immediate use for it. If I could achieve a fairly true binaural effect that could synchronize/etc sounds (for now with spatially correct sonic emitters in the 2.79 game engine would do!) then I could PreVis the audio effect of something I have in mind that could be physically produced. That could be wonderful so of course i have to remain discreet.

My background is theatrical sound and video design and in 2001 I used a Umax (a rare Mac clone) to previs a projection design on a simple model of the set I reproduced in Bryce 3D. Got an award. I started with Infini-D years before and quickly set up two cameras and two windows so I had a real time stereo view into the world of mesh I was manipulating, it being all about 3D and all it made sense to me. Stereoscopy is also something I’ve done a bit of innovating with and naturally enough Binaural Audio is something I’m eager to be experimental with in this wild world of Blender.

Thanks for asking, hope I get to see the effort developed. Here’s “1+1=4D: Stereoscopic Video in Real Time” for your pleasure. It was eventually turned into an exhibit at the Exploratorium in San Francisco, after I was gone.

Personally, ever since I saw the video from 2 minute papers about it, I’ve been hoping that Blender would get it some day. https://www.youtube.com/watch?v=DzsZ2qMtEUE
Sound path tracing looks to be much faster than normal path tracing, as in the video they can get some very nice results with just ~100k sound samples - for reference, a 1080p image at just 1spp is ~21 times more samples. (or ~670 times more for 32spp)

I don’t like Nvidia’s solution as it’s proprietary, I don’t have an Nvidia GPU and others don’t either yet one is needed and when the people in the video managed to get the sound at realtime speeds on a single thread of a probably very slow “3.5ghz processor” then I don’t see why Blender should use that proprietary implementation. Nvidia’s thing is probably meant for game devs that don’t have a path tracer.

Blender already has a light path tracer that works with any gpu or cpu so I hope making it work with sound would not be too hard.

2 Likes

Isn’t it because a 1080p image is made of 2.073.600 pixels?

Yes, that’s how I got the value. We render images with massive sample counts, but for sound much fewer seem to be enough.

I began looking for solutions to this last year - this thread is still one of the few robust conversations on the subject that I can find. I think this feature would invite novel compositional strategies for both video and audio, given the extensive use of drivers throughout Blender - not to mention the node-based workflows that are making steady progress.

Personally, I think we should be pursuing this kind of novelty in FOSS spaces - proprietary software does well enough to serve existing industry needs but its peer production that can yield innovation that industry will never demand, or so the argument goes. I think Blender is doing this with the grease pencil tools - relevant to industry but not something that established studios would have necessarily been in a position to demand from the whatever software comprises their existing workflow. But you never really know where the insight will break through, so we have to take advantage of the opportunity to move in every direction according to our own needs and interests.

It is good to try to get a read on things like this but I would encourage you to consider that some aspects of your proposal might yield things that you haven’t thought of yet and that those might end up being what take off. At any rate, even matching what SoundScape Renderer gives us would be an upgrade that many may have to see in order to recognize its value. Perhaps there’s a way to start with an addon?

This is a topic I’m very interested in. I guess the question is do we want “accurate” audio, or “believable” audio.

From a brief survey, if we want just “believable” object based audio there are a couple of options:

VISR which provides a python API
Steam audio which provides plugins to Unity and Unreal, and supports embree/AMD (OpenCL 1.2) acceleration. It provides a C API as well.

Both are open source.

The VISR project hasn’t been updated in two years, but the steam audio project is still actively maintained.

Edit:
Also there is i-simpa which is also open source, and supports python scripting.

There is also the “Resonance” plugin by Google, which seems to still be actively developed.

Here is a fairly old comparison between Steam Audio, Resonance, and the closed source facebook version: https://designingsound.org/2018/03/29/lets-test-3d-audio-spatialization-plugins/

As far as I can tell from the documentation, neither Steam Audio nor Resonance compute the ITD (interaural time difference). In practice the delay for ears separated by 20cm is around half a millisecond in the worst case.

It seems like Steam audio makes the further simplifies the model by not accounting for the propagation of sound, but Resonance does. Please correct me if I am wrong.