1aSC13. Neural processes of audio--visual speech perception.

Session: Monday Morning, December 2

Time:


Author: Satoshi Imaizumi
Location: RILP, Univ. Tokyo, Bunkyo-ku, Tokyo, 113 Japan
Author: Koichi Mori
Location: RILP, Univ. Tokyo, Bunkyo-ku, Tokyo, 113 Japan
Author: Shigeru Kiritani
Location: RILP, Univ. Tokyo, Bunkyo-ku, Tokyo, 113 Japan
Author: Masato Yumoto
Location: RILP, Univ. Tokyo, Bunkyo-ku, Tokyo, 113 Japan
Author: Hideaki Seki
Location: Chiba Inst. Tech., Japan

Abstract:

Neural processes related to audio--visual speech perception were investigated by measuring the mismatch magnetic fields (MMF), which reflect a neural activity detecting deviant stimuli randomly inserted in a stream of rapidly repeating frequent stimuli. Three audio--visual stimuli were used: AbVg (audio signal /ba/ with discrepant visual signal /ga/), AbVb, and AdVd. MMF were measured from the left hemisphere of 16 normal hearing subjects using a 37 ch SQUID magnetometer. The rate of non-/ba/ responses to AbVg (McGurk fusion effect) varied depending on the stimulus condition and subjects. It was 80% when AbVg was the frequent, but was 45% when AbVg was the deviant. Significant MMF-like fields were excited in the auditory cortex by deviant AbVg embedded in frequent AbVb. The subjects with a low-fusion rate had larger MMF-like fields than those with a high rate. These results suggest that the auditory mismatch detection process is affected by visual signal, and phonetic categorization is affected by a module which can either fuse or dissociate audio--visual information.


ASA 132nd meeting - Hawaii, December 1996