Re: Auditory-visual comodulation (grant )

Subject: Re: Auditory-visual comodulation
From:    grant  <grant(at)NICOM.COM>
Date:    Fri, 3 Apr 1998 12:56:15 -0500

This is a multi-part message in MIME format.
--------------C385372B13CCDA95623DE1D9
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit

BRUNO H. Repp wrote:

> These are very interesting results! However, I disagree that there were
 "problems"
> with our study that led us not to find the "correct" results. The aims of our
> study were different. We did not investigate the effect of comodulation of
 visual
> and auditory input on detectability. Rather, we were interested in effects of
 the
> lexical status of the words to be detected. Effects of lexicality are
 definitely
> top-down, whereas comodulation effects of the sort that Ken has demonstrated
 are
> arguably bottom-up, even though they require cross-modal integration of some
 kind.
> In our study, as Ken has pointed out, there was always a considerable degree
 of
> comodulation between auditory and visual inputs. Also, our use of
> signal-correlated noise was quite deliberate and not a "problem". Only the
> presence of some inaccuracies in temporal alignment may be considered a
> shortcoming of our study. However, the degree of synchrony present was
 sufficient
> to lead to very clear lexical bias effects, whereas there was no effect on
> sensitivity to the presence of speech in (speech-like) noise. It seems
 unlikely to
> me that there would have been an effect on sensitivity if the synchrony had
 been
> more accurate. Does the enhancement effect that Ken has demonstrated go away
 when
> auditory and visual inputs are misaligned by +/-50 ms? I would be surprised if
> that were the case. His "mismatched" lipread information, I suppose, is
 completely
> out of synch with the auditory (masked) input.

In the original article by Repp, Frost, and Zsiga (1992) the stated goal (as
reported in the abstract) was to "...investigate whether simultaneous
 speechreading
can influence the detection of speech in envelope-matched noise.". The results
 of
the first experiment showed that "Speech detection was not facilitated by an
audio-visual match, which suggests that listeners relied on low-level auditory
 cues
whose perception was immune to cross-modal top-down influences.".

I have argued that the use of envelope-matched noise and the inability to ensure
audio-visual stimulus alignments prevented Repp et al from observing the
 influence
of visual cues on auditory detection. The current experiment underway in our
laboratory shows that visible movements of the lips that match an acoustic
 stimulus
do indeed influence auditory detection.

The second question of lexical mediation addressed in the Repp et al study is,
 in my
opinion, still left unanswered. In fact, I would suggest that the visual
 matching
signal provides useful temporal and possibly spectral information (in the case
 of a
lipread signal) that can inform the auditory system when in time and where in
frequency to listen. This may be a general property of any audio-visual matching
stimulus and need not be speech. For example, the visual signal may be a
 bouncing
ball, and the acoustic signal may be any dynamically changing stimulus. At this
point we are not sure what stimulus conditions are required to observe the
 effect.
My guess is that any combination of A and V comodulation will do the trick and
 that
by manipulating the degree of audio-visual comodulation we could modulate the
strength of the release from masking.

Ken

--------------C385372B13CCDA95623DE1D9
Content-Type: text/x-vcard; charset=us-ascii; name="vcard.vcf"
Content-Transfer-Encoding: 7bit
Content-Description: Card for Ken W. Grant
Content-Disposition: attachment; filename="vcard.vcf"

begin:          vcard
fn:             Ken W. Grant
n:              Grant;Ken W.
org:            Army Audiology & Speech Center
adr;dom:        Research Section;;Walter Reed Army Medical Center;;Washington,
 DC;20307-5001;
email;internet: grant(at)nicom.com
title:          Research Audiologist
tel;work:       (202) 782-8596
tel;fax:        (202) 782-9228
x-mozilla-cpt:  ;0
x-mozilla-html: FALSE
version:        2.1
end:            vcard


--------------C385372B13CCDA95623DE1D9--