Subject: Re: perceptual audio evaluation tests From: Christian Uhle <christian.uhle@xxxxxxxx> Date: Wed, 17 Dec 2014 12:59:19 +0100 List-Archive:<http://lists.mcgill.ca/scripts/wa.exe?LIST=AUDITORY>What do you mean by "quality is not necessarily transitive"? Christian Uhle schrieb: > > What do you mean > > James Johnston schrieb: >> The problem I have with MUSHRA is that quality is not a 1-dimensional >> quantity, and quality is not necessarily transitive, either. >> >> On Wed, Dec 17, 2014 at 12:13 AM, Christian Uhle >> <christian.uhle@xxxxxxxx >> <mailto:christian.uhle@xxxxxxxx>> wrote: >> >> Dear Mark, >> >> I'm not aware of a standardized method for subjective testing >> (i.e. listening tests) for source separation. >> I recommend a testing similar to MUSHRA, i.e. a multiple stimuli >> test with hidden reference and anchor. >> >> For the reference (and hidden reference) the best option IMO is to >> start with signals where the desired signal and the interfering >> signals are separately available. The reference signal is then >> either the desried signal or as a mixture of the desired and >> interfering signal where the interferer is attenuated (in case >> your separation does not aim at complete separation but at an >> enhancement of the desired signal w.r.t the interferer). >> >> For the anchor: the standard 3.5 kHz low-pass filtered signal is >> one option. Of course, the other conditions (aka processed >> signals) should not sound much worse than the anchor. So, the >> processing for deriving the anchor signals depends a bit on the >> conditions under test. Starting with an oracle mask, introducing >> degradations to it and computing an output signal is one option >> when testing BSS methods that are based on spectral weighting. >> Also, having more interefer in the anchor thn in the conditions >> under test might be good. >> >> The main problem IMO is this: >> when using a MUSHRA test for accessing the quality of let's say an >> audio codec, we often ask for transparency. This is a >> one-dimensional quantity. >> Evaluation of BSS is about a multi-dimensional quantity: 1) >> reducing the interference and 2) sound quality are the most >> important dimensions here. >> You can either ask the test listeners for a combined rating (in a >> preference test) or you ask for ratings regarding each of the >> characteristics separately. >> This depends a bit on the aim of the test (e.g. an aim could be >> comparing different methods in order to decide which one to buy, >> or testing during development for the purpose of tuning). >> >> Good luck, >> Christian >> >> Mark Cartwright schrieb: >> >> Hello, >> >> I'm looking for the list's opinions on perceptual audio >> evaluation listening tests for signals that have large >> impairments. In particular, I'm primarily interested in the >> evaluation of the output of source separation algorithms. What >> standardized tests do people recommend (e.g. ITU-R BS.1534-2 / >> MUSRHA, ITU-T P.800, etc.) and what are their pros and cons? >> Also, are there other tests that are preferred over these but >> have not yet been standardized? >> >> Thanks! >> >> Mark >> >> -- Mark Cartwright <mcartwright@xxxxxxxx >> <mailto:mcartwright@xxxxxxxx> >> <mailto:mcartwright@xxxxxxxx >> <mailto:mcartwright@xxxxxxxx>>> >> PhD Candidate in Computer Science >> Interactive Audio Lab <http://music.cs.northwestern.edu/>, >> Northwestern University >> www.markcartwright.com <http://www.markcartwright.com> >> <http://www.markcartwright.com/> >> >> >> >> -- Dr.-Ing. Christian Uhle >> Senior Scientist >> AudioLabs-IIS >> Fraunhofer Institute for Integrated Circuits IIS >> >> Am Wolfsmantel 33 >> 91058 Erlangen >> Germany >> E-mail: christian.uhle@xxxxxxxx >> <mailto:christian.uhle@xxxxxxxx> >> Phone: +49(0) 9131 / 776 - 6230 >> Fax: +49(0) 9131 / 776 - 6099 >> >> http://www.iis.fraunhofer.de/amm/ >> http://www.audioblog.iis.fraunhofer.com >> >> Meet us at these events: >> >> CES Las Vegas, January 6-9, 2015 >> MWC Barcelona, March 2-5, 2015 >> NAB Las Vegas, April 13-16, 2015 >> ... >> >> >> >> -- >> James D. (jj) Johnston >> Independent Audio and Electroacoustics Consultant > >