Most people hear 'baba' with their eyes closed, and 'dada' or 'gaga' with their eyes open. The effect is created by playing a 'baba' audio track dubbed onto a visual track of 'gaga'. The McGurk effect* provides an excellent example of the importance of associations for many of the computations the brain performs, including speech recognition. Why is what we 'hear' influenced by what we see? Every time you heard someone say a word with the letter 'b' in it, while you were looking at them, you saw their lips come together and separate. This has not slipped the notice of your cortical circuits. Thus the sound of 'ba' together with the sight of 'ba' (lips touching and separating) have become intertwined in your brain. So much so that your brain refuses to believe that if it does not see the lips touch and separate that it heard 'ba'. Phonetically, 'd' and 'g' are similar to 'b' because they are all unaspirated phonemes, but 'd' and 'g' are enunciated without closing of the mouth. Thus your brain often deals with the conflicting auditory and visual information by settling for 'da'.

The association between the sound and sight of 'ba' is so strong, that it simply does not make sense to your sensory system that one might be occurring without the other. It is like seeing a face without a nose - in the absence of a nose your brain has trouble classifying an image as a face. One notable exception to the fact that the sound 'ba' is almost always accompanied by the lips separating occurs when we watch dubbed movies - particularly badly dubbed Chinese Kung-fu movies. Thus a prediction emerges is that someone who spent way to much time watching badly dubbed movies might not experience the McGurk effect.

Although I'm using the McGurk effect to highlight the associative architecture of the brain, it is typically used as an example of cross-modal integration, that is, that our subjective perception through one sense is altered by other senses, in this hearing is function of not only what comes in through the ears but through the eyes. This cross-modal integration, however, relies on the brains ability to detect and record associated events.

Special thanks to Lisa for her help with this demo.

1 Gurk H, MacDonald J (1976) Hearing lips and seeing voices. Nature 264:746-748.