correlation of vision and audition (fwd)

[Al Bregman <bregman@xxxxxxxxxxxxxxxxxxxx>]

Dear Colleagues,

A doctoral student in the music recording program at McGill, who is taking
my course in auditory perception, wrote me to ask about a phemonenon that
he had observed while doing animation.  I include a part of his message:

---------- Forwarded message ----------
Date: Fri, 6 Feb 1998 14:15:35 -0500 (EST)
From: Jason Corey <corey@music.mcgill.ca>

While creating and editing a soundtrack for an animated film, it became
apparent that sounds occurring sychronously in time with visual events on
the screen, had a effect on how I perceived the visuals.  For one
particular scene there happened to be a great deal of activity happening
in the animation, with different events happening at different positions
on the screen.  Without a sound track there were many events that were not
perceived, until a sound effect was sychronised with the particular visual
events.

It seems that by having sound accompany a visual, many more details
of the visual are perceived than without a soundtrack.  It would seem
that our auditory system helps the visual system to "sort out" the
details when there are many events happening visually.  Maybe this
somehow helps us focus on certain visual details more than others.  My
guess is that by changing the soundtrack to a given moving picture, it is
possible to alter what the subject will focus on.

I was wondering how much work has been done in this area.  Any help would
be appreciated.
 ...

Jason Corey
--------- end of forwarded message ---------------


I know about the work on the ventriloquism effect by Thomas, 1941, by
Witkin et al, 1952, and by Bertelson, Radeau and others more recently, but
this research is more concerned with the PERCEIVED LOCATION of the sound
or visual object rather than its behavior over time.  I think the question
is about how a correlated sound helps in the visual parsing of a display.

Concerning the reverse, how a correlated visual event would help in the
parsing of an auditory presentation, we have probably all seen one person
using hand motions in the air to direct another person's attention to a
particular part in a complicated piece of music.

I am looking for information about these phenomena - both when audition
assists visual parsing or the reverse.

I know about some of the research that looks at the role of visual cues in
the perception of speech (Dodd, 1977, 1979, 1980; Massaro, 1987; McGurk
and colleagues).  I also am aware of some research on the influence of
correlated sounds on apparent motion (phi) by Gilbert (1939) and by
O'Leary and Rhodes (1984).  There is also the work of Spelke and her
colleagues on the integration of correlated visual and auditory events by
infants.

I have also heard of a study by Sumby and Pollack on how the seeing of a
speaker can help the listener to understand speech in noise, but I
don't have the exact reference.

The effects of visual input on speech understanding may be entirely due to
the fact that the visual input actually supplies additional
information about the identity of the speech signal.  Has anybody
determined whether any part of the effect may be in assisting the listener
to parse the mixture of sounds that might be present?

Does anyone know of articles, chapters or technical reports specifically
on the issue of how information from one modality helps us to parse
mixtures in another?  Or of the exact reference to the Sumby-Pollack
article?

If so, I'd appreciate hearing about them.

Thanks,

Al

----------------------------------------------------------------------
Albert S. Bregman,  Professor,  Dept of Psychology,  McGill University
1205  Docteur Penfield Avenue,   Montreal,  Quebec,  Canada   H3A 1B1.
Phone: +1 514-398-6103 Fax: -4896  Email: bregman@hebb.psych.mcgill.ca
Lab Web Page: http://www.psych.mcgill.ca/labs/auditory/laboratory.html
----------------------------------------------------------------------