[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: identification test procedure
At 10:43 AM 10/9/98 -0500, James W. Beauchamp wrote:
>Folks,
>
>A colleague of mine is using a particular procedure to test how
well
>synthetic piano tones resemble "original" piano tones.
The "original" tones
>are actually analyzed and resynthesized using a phase vocoder
method (in
>order to eliminate background noise) and the synthetic tones
are
>resynthesized with a fair amount of data reduction. Pitches
between E1 and
>D7 of equal numbers of original and synthetic tones are
presented in random
>order to the listener, and the listener's task is to choose
which tones are
>synthetic. My colleague's contention is that if the listener
were to simply
>guess, the score would be 50%, and that any score less than 50%
indicates
>poor ability to distinguish, thus proving the efficacy of the
data
>reduction method. Indeed, the scores ranged from 15% to
45%.
>
>My question is -- Does this really work? If the listener tries
to maximize
>his/her score, then he/she might tend to guess, moving the score
towards
>50%. On the other hand, if the subject is truly honest and
he/she cannot
>distinguish, he/she would choose no tones and would score 0%. I
guess it
>would depend on how the subjects are instructed to make their
decisions,
>but then can you trust them to always follow instructions?
Shouldn't the
>false positives also be reported?
>
>Is there a way to properly evaluate the results of this
test?
>
>What is the best paradigm to test the efficacy of a data
reduction method?
>
>Jim Beauchamp
>University of Illinois at Urbana-Champaign
>j-beauch@uiuc.edu
>
>McGill is running a new version of LISTSERV (1.8d on Windows
NT).
>
Jim,
The biggest problem with this method is that it does not distinguish
between discrimination and response bias. Try to use a confidence
rating-scale ROC method (described at length in the Green and Swets
Signal Detection Theory and Psychophysics book [1966], among other
places). Actually, you are trying to evaluate the null hypothesis that
the d' for discriminating the two pianos will be negative, which is
considered hairy by many psychophysicists. (A negative d' is as good an
indicator of discriminability as the positive, except that the listener
is standing on his/her head.) Why don't you just tell them that they will
hear two pianos, A and B, and ask them later whether they thought A or B
to be the "true" instrument? Psychophysically speaking, this
would lead to a cleaner experiment.
Pierre
****************************************************************************
Pierre Divenyi
Experimental
Audiology Research (151)
V.A. Medical Center, Martinez, CA 94553, USA
Phone: (925) 370-6745
Fax: (925) 228-5738
E-mail :
pdivenyi@marva4.ebire.org
****************************************************************************