correlation of vision and audition (fwd) (Al Bregman )


Subject: correlation of vision and audition (fwd)
From:    Al Bregman  <bregman(at)hebb.psych.mcgill.ca>
Date:    Fri, 6 Feb 1998 18:27:54 -0500

Dear Colleagues, A doctoral student in the music recording program at McGill, who is taking my course in auditory perception, wrote me to ask about a phemonenon that he had observed while doing animation. I include a part of his message: ---------- Forwarded message ---------- Date: Fri, 6 Feb 1998 14:15:35 -0500 (EST) From: Jason Corey <corey(at)music.mcgill.ca> While creating and editing a soundtrack for an animated film, it became apparent that sounds occurring sychronously in time with visual events on the screen, had a effect on how I perceived the visuals. For one particular scene there happened to be a great deal of activity happening in the animation, with different events happening at different positions on the screen. Without a sound track there were many events that were not perceived, until a sound effect was sychronised with the particular visual events. It seems that by having sound accompany a visual, many more details of the visual are perceived than without a soundtrack. It would seem that our auditory system helps the visual system to "sort out" the details when there are many events happening visually. Maybe this somehow helps us focus on certain visual details more than others. My guess is that by changing the soundtrack to a given moving picture, it is possible to alter what the subject will focus on. I was wondering how much work has been done in this area. Any help would be appreciated. ... Jason Corey --------- end of forwarded message --------------- I know about the work on the ventriloquism effect by Thomas, 1941, by Witkin et al, 1952, and by Bertelson, Radeau and others more recently, but this research is more concerned with the PERCEIVED LOCATION of the sound or visual object rather than its behavior over time. I think the question is about how a correlated sound helps in the visual parsing of a display. Concerning the reverse, how a correlated visual event would help in the parsing of an auditory presentation, we have probably all seen one person using hand motions in the air to direct another person's attention to a particular part in a complicated piece of music. I am looking for information about these phenomena - both when audition assists visual parsing or the reverse. I know about some of the research that looks at the role of visual cues in the perception of speech (Dodd, 1977, 1979, 1980; Massaro, 1987; McGurk and colleagues). I also am aware of some research on the influence of correlated sounds on apparent motion (phi) by Gilbert (1939) and by O'Leary and Rhodes (1984). There is also the work of Spelke and her colleagues on the integration of correlated visual and auditory events by infants. I have also heard of a study by Sumby and Pollack on how the seeing of a speaker can help the listener to understand speech in noise, but I don't have the exact reference. The effects of visual input on speech understanding may be entirely due to the fact that the visual input actually supplies additional information about the identity of the speech signal. Has anybody determined whether any part of the effect may be in assisting the listener to parse the mixture of sounds that might be present? Does anyone know of articles, chapters or technical reports specifically on the issue of how information from one modality helps us to parse mixtures in another? Or of the exact reference to the Sumby-Pollack article? If so, I'd appreciate hearing about them. Thanks, Al ---------------------------------------------------------------------- Albert S. Bregman, Professor, Dept of Psychology, McGill University 1205 Docteur Penfield Avenue, Montreal, Quebec, Canada H3A 1B1. Phone: +1 514-398-6103 Fax: -4896 Email: bregman(at)hebb.psych.mcgill.ca Lab Web Page: http://www.psych.mcgill.ca/labs/auditory/laboratory.html ----------------------------------------------------------------------


This message came from the mail archive
http://www.auditory.org/postings/1998/
maintained by:
DAn Ellis <dpwe@ee.columbia.edu>
Electrical Engineering Dept., Columbia University