Music of Sound

I dont use that interweb expression LOL much/if at all, but I am about to – simply because this actually made me laugh out loud as I observed my brain & perceptions do a little twist inside out. If you have never heard this effect then you may be about to experience the same.

Synthesis via resynthesis is a fairly technical but straight forward process of taking a naturally occurring sound, analysing it & then re-creating it using many sine waves at different pitches. When it is done very high resolution (eg with a Kyma system) it effectively gives you the ability to ‘play’ the spectrum of a sound, but I have never considered applying the approach to dialogue, until now…

Matt Davis of the MRC Cognition & Brain Sciences Unit at Cambridge has a thorough article here about the process & how our brain interprets it, but to illustrate the cognitive effect clearly, listen to this:

Audio clip: Adobe Flash Player (version 9 or above) is required to play this audio clip. Download the latest version here. You also need to have JavaScript enabled in your browser.

sounds like R2D2 right? now listen to this:

Audio clip: Adobe Flash Player (version 9 or above) is required to play this audio clip. Download the latest version here. You also need to have JavaScript enabled in your browser.

ok, now play sine speech 1 again – notice any perceptual shift between now & ten seconds prior? I laughed out loud… sorry LOL’d

Heres a few other examples:

Audio clip: Adobe Flash Player (version 9 or above) is required to play this audio clip. Download the latest version here. You also need to have JavaScript enabled in your browser.

Audio clip: Adobe Flash Player (version 9 or above) is required to play this audio clip. Download the latest version here. You also need to have JavaScript enabled in your browser.

got it?

Audio clip: Adobe Flash Player (version 9 or above) is required to play this audio clip. Download the latest version here. You also need to have JavaScript enabled in your browser.

Audio clip: Adobe Flash Player (version 9 or above) is required to play this audio clip. Download the latest version here. You also need to have JavaScript enabled in your browser.

how about that one?

Audio clip: Adobe Flash Player (version 9 or above) is required to play this audio clip. Download the latest version here. You also need to have JavaScript enabled in your browser.

Audio clip: Adobe Flash Player (version 9 or above) is required to play this audio clip. Download the latest version here. You also need to have JavaScript enabled in your browser.

any different?

Audio clip: Adobe Flash Player (version 9 or above) is required to play this audio clip. Download the latest version here. You also need to have JavaScript enabled in your browser.

Audio clip: Adobe Flash Player (version 9 or above) is required to play this audio clip. Download the latest version here. You also need to have JavaScript enabled in your browser.

A valuable reminder of the things we take for granted….

Basic info here

I know, it sounds like it should be related to the MacGuffin, but it isnt… Whether they know the name or not, the McGurk Effect is very well known to most ADR Editors eg when a director wants to change an onscreen line & tries to put different words in the mouth of an actor… No matter how well performed & edited the new line is, your brain just doesnt accept it as true/real/honest, and that is dangerous territory to be in for any film maker as it can abruptly destroy suspension of disbelief, or even worse just slowly undermine it…. As a very simple example watch this video & take particular note of what the actor is saying:

direct youtube link for iphone/ipod touch

Ok now play the clip again, but shut your eyes & listen with your ears only… See?

The McGurk Effect also explains why I would much rather read subtitles on a film than have it dubbed into english – no matter how good the sync is!

Thanks to Carl for the link!