Dear Mark,
I'm not aware of a standardized method for subjective testing (i.e. listening tests) for source separation.
I recommend a testing similar to MUSHRA, i.e. a multiple stimuli test with hidden reference and anchor.
For the reference (and hidden reference) the best option IMO is to start with signals where the desired signal and the interfering signals are separately available. The reference signal is then either the desried signal or as a mixture of the desired and interfering signal where the interferer is attenuated (in case your separation does not aim at complete separation but at an enhancement of the desired signal w.r.t the interferer).
For the anchor: the standard 3.5 kHz low-pass filtered signal is one option. Of course, the other conditions (aka processed signals) should not sound much worse than the anchor. So, the processing for deriving the anchor signals depends a bit on the conditions under test. Starting with an oracle mask, introducing degradations to it and computing an output signal is one option when testing BSS methods that are based on spectral weighting.
Also, having more interefer in the anchor thn in the conditions under test might be good.
The main problem IMO is this:
when using a MUSHRA test for accessing the quality of let's say an audio codec, we often ask for transparency. This is a one-dimensional quantity.
Evaluation of BSS is about a multi-dimensional quantity: 1) reducing the interference and 2) sound quality are the most important dimensions here.
You can either ask the test listeners for a combined rating (in a preference test) or you ask for ratings regarding each of the characteristics separately.
This depends a bit on the aim of the test (e.g. an aim could be comparing different methods in order to decide which one to buy, or testing during development for the purpose of tuning).
Good luck,
Christian
Mark Cartwright schrieb:
Hello,
I'm looking for the list's opinions on perceptual audio evaluation listening tests for signals that have large impairments. In particular, I'm primarily interested in the evaluation of the output of source separation algorithms. What standardized tests do people recommend (e.g. ITU-R BS.1534-2 / MUSRHA, ITU-T P.800, etc.) and what are their pros and cons? Also, are there other tests that are preferred over these but have not yet been standardized?
Thanks!
Mark
--
Mark Cartwright <mcartwright@u.northwestern.edu <mailto:mcartwright@u.northwestern.edu>>
PhD Candidate in Computer Science
Interactive Audio Lab <http://music.cs.northwestern.edu/>, Northwestern University
www.markcartwright.com <http://www.markcartwright.com/>
--
Dr.-Ing. Christian Uhle
Senior Scientist
AudioLabs-IIS
Fraunhofer Institute for Integrated Circuits IIS
Am Wolfsmantel 33
91058 Erlangen
Germany
E-mail: christian.uhle@iis.fraunhofer.de
Phone: +49(0) 9131 / 776 - 6230
Fax: +49(0) 9131 / 776 - 6099
http://www.iis.fraunhofer.de/amm/
http://www.audioblog.iis.fraunhofer.com
Meet us at these events:
CES Las Vegas, January 6-9, 2015
MWC Barcelona, March 2-5, 2015
NAB Las Vegas, April 13-16, 2015
...