BGE - Get audio spectrum/waveform to make lipsync.

Game Engine Game Engine Support and Discussion
1

Hello guys!

I’m an advanced programmer and game programmer, but I’m completly newbie with any kind of sound. So, I’m looking for a way to play a sound on BGE (using python) and get the spectrum + the current time of the sound… Or some idea to help me to recognize what is going on in the sound to program a lip sync method.

I’ve in mind how I’ll program this (the lip sync) this is not what I’m asking here, but I’ve to know a little bit more about the sound player on BGE…

Thanks!

1 Like
2

Actually there is no built in FFT in Blender (except getSpectrum, but it needs fmod to be installed and used).
You can use numpy or other libraries to get it working.

https://blenderartists.org/forum/showthread.php?359877-Real-Time-Lip-Sync-in-BGE (download down, https://github.com/dhall5/Blender-Game-Engine-Lip-Sync, https://www.youtube.com/watch?v=i_oaXKyNAgM)

https://blenderartists.org/forum/showthread.php?390591-real-time-audio-frequency-spectrum

3

I actually used python wave module to make something like this:


import wave
from array import array
from bge import logic


bands = 100 #number of lines
accuracy = 1


scene = logic.getCurrentScene()


def bounce(cont):
    own = cont.owner
    
    if "init" not in own:
        own["init"] = True
        own["pos"] = own.worldPosition.x.copy()
        
    s = list(logic.amp)
    dynamic = abs(sum(s)/(len(s)**2.5))*2
        
    desired = abs(logic.amp[-int(own.worldPosition.x*1/accuracy)-1])/100/(own["pos"]/5)+dynamic
    
    z = own.localScale.y
    
    if z < desired:
        own.localScale.y += (desired-z)/25
    elif z > desired:
        own.localScale.y -= (z-desired)/25


    
def start(cont):
    own = cont.owner


    for i in range(1,bands+1):
        own.worldPosition.x = i*2
        newObj = scene.addObject("Cube", own, 0) #The cube in a hidden layer will have an always sensors
                                                 #with true pulse mode eabled running the bounce function above^^^


def amplitude(own, wav, t):
    
    frames = wav.getnframes()
    rate = wav.getframerate()
    head = int(t * rate)
    delta = head - wav.tell()
    
    if delta < 0:
        delta = head
        wav.rewind()
        
    chunk = array('h', wav.readframes(delta))
        
    return chunk if chunk else [1,1,1]


   
def analyse(cont): #Use a master object for this. Must have pulse mode running it.
    own = cont.owner


    s = cont.actuators["Sound"]


    if not "init" in own:


        music = own["music"] #put the file of the song in a string.
        
        if not music.endswith(".wav"):
            
            import subprocess


            subprocess.call(logic.expandPath("//")+"ffmpeg.exe -y -i \""+music+"\" \""+music.split(".")[0]+".wav\"") #convert the non wav sound to wav, using ffmpeg.
            own["music"] = music.split(".")[0]+".wav"
            music = own["music"]
            
        own["init"] = True
        s.sound = aud.Factory.file(music)
        s.mode = 4
        
        cont.activate(s)


        logic.wav = wave.open(music)
        logic.power = 1


    logic.soundTime = s.time
    logic.amp = amplitude(own, logic.wav, logic.soundTime)

It should work probably with a bit of jiggling around perhaps because I modified it to be simpler than what I used.

1 Like
4

Very intersting example, now, I have a point to start learning about this. Thalnk you guys! :smiley:

1 Like