[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: perceptual audio evaluation tests
Dear Mark,
I'm not aware of a standardized method for subjective testing (i.e.
listening tests) for source separation.
I recommend a testing similar to MUSHRA, i.e. a multiple stimuli test
with hidden reference and anchor.
For the reference (and hidden reference) the best option IMO is to start
with signals where the desired signal and the interfering signals are
separately available. The reference signal is then either the desried
signal or as a mixture of the desired and interfering signal where the
interferer is attenuated (in case your separation does not aim at
complete separation but at an enhancement of the desired signal w.r.t
the interferer).
For the anchor: the standard 3.5 kHz low-pass filtered signal is one
option. Of course, the other conditions (aka processed signals) should
not sound much worse than the anchor. So, the processing for deriving
the anchor signals depends a bit on the conditions under test. Starting
with an oracle mask, introducing degradations to it and computing an
output signal is one option when testing BSS methods that are based on
spectral weighting.
Also, having more interefer in the anchor thn in the conditions under
test might be good.
The main problem IMO is this:
when using a MUSHRA test for accessing the quality of let's say an audio
codec, we often ask for transparency. This is a one-dimensional quantity.
Evaluation of BSS is about a multi-dimensional quantity: 1) reducing the
interference and 2) sound quality are the most important dimensions here.
You can either ask the test listeners for a combined rating (in a
preference test) or you ask for ratings regarding each of the
characteristics separately.
This depends a bit on the aim of the test (e.g. an aim could be
comparing different methods in order to decide which one to buy, or
testing during development for the purpose of tuning).
Good luck,
Christian
Mark Cartwright schrieb:
Hello,
I'm looking for the list's opinions on perceptual audio evaluation
listening tests for signals that have large impairments. In
particular, I'm primarily interested in the evaluation of the output
of source separation algorithms. What standardized tests do people
recommend (e.g. ITU-R BS.1534-2 / MUSRHA, ITU-T P.800, etc.) and what
are their pros and cons? Also, are there other tests that are
preferred over these but have not yet been standardized?
Thanks!
Mark
--
Mark Cartwright <mcartwright@xxxxxxxxxxxxxxxxxx
<mailto:mcartwright@xxxxxxxxxxxxxxxxxx>>
PhD Candidate in Computer Science
Interactive Audio Lab
<http://music.cs.northwestern.edu/>, Northwestern University
www.markcartwright.com <http://www.markcartwright.com/>
--
Dr.-Ing. Christian Uhle
Senior Scientist
AudioLabs-IIS
Fraunhofer Institute for Integrated Circuits IIS
Am Wolfsmantel 33
91058 Erlangen
Germany
E-mail: christian.uhle@xxxxxxxxxxxxxxxxx
Phone: +49(0) 9131 / 776 - 6230
Fax: +49(0) 9131 / 776 - 6099
http://www.iis.fraunhofer.de/amm/
http://www.audioblog.iis.fraunhofer.com
Meet us at these events:
CES Las Vegas, January 6-9, 2015
MWC Barcelona, March 2-5, 2015
NAB Las Vegas, April 13-16, 2015
...