Re: [AUDITORY] Fwd: [AUDITORY] MUSHRA test with open reference; and then without open reference (Jonathan James Digby )

Subject: Re: [AUDITORY] Fwd: [AUDITORY] MUSHRA test with open reference; and then without open reference From: Jonathan James Digby <digbyphonic@xxxxxxxx> Date: Fri, 23 Aug 2024 08:04:15 +0100 This is a multipart message in MIME format. ------=_NextPart_000_0093_01DAF533.0C80B630 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Dear all, =20 I=92m interested to hear the cohort=92s opinion on the suitability of = these parametric tests based upon the need to transform the data, e.g. = attempting to adjust for rating variance. Any question of reliability is surely amplified by the unacceptably low number of participants in many (most) listening experiments? =20 Mendon=E7a, C. and Delikaris-Manias, S. (2018) =91Statistical tests with = MUSHRA data=92, in Audio engineering society convention 144. Available at: http://www.aes.org/e-lib/browse.cfm?elib=3D19402 (Accessed: 23 August = 2024). =20 Thank you for the discussion. =20 ___________________________________________ Jonathan J Digby Audio Engineer; Digbyphonic Ltd Postgraduate Researcher; University of Derby, GBR Tel: +44 7833 151 181 =20 From: Thomas Biberger <thomas.biberger@xxxxxxxx>=20 Sent: Thursday, August 22, 2024 7:08 AM Subject: AW: Fwd: [AUDITORY] MUSHRA test with open reference; and then without open reference =20 Dear Arijit et al., =20 a study that goes in this direction was recently carried out in = Oldenburg by Henning Schepker, Florian Denk, Birger Kollmeier and Simon Doclo. They investigated acoustic transparency in hearables and for this they made = a perceptual sound quality evaluation of commercially available hearables = with 20 normal-hearing listeners. In their 1st Experiment an explicit = reference was provided to the participants, while in the 2nd Experiment no = explicit reference was provided.=20 =20 In their conclusions they state that "a comparison of the discrimination = and reliability of the quality ratings of both experiments showed that participants are generally more reliable and can discriminate better = when a reference signal is provided." =20 For more details, see =20 H. Schepker, F. Denk, B. Kollmeier, and S. Doclo, =93Acoustic = Transparency in Hearables=96Perceptual Sound Quality Evaluations=94 J. Audio Eng. Soc., vol. 68, no. 7/8, pp. 495=96507, (2020 = July/August.). DOI: https://doi.org/10.17743/jaes.2020.0045 =20 Hope that helps! =20 BR, Thomas =20 -- =20 Dr. Thomas Biberger =20 Medizinische Physik and Cluster of Excellence Hearing4all Psychoacoustics & Auditory Modelling Universit=E4t Oldenburg D-26111 Oldenburg Germany =20 phone:+49-441-798 3528 room: W30-3-330 email: thomas.biberger@xxxxxxxx <mailto:thomas.biberger@xxxxxxxx>=20 =20 Von: AUDITORY - Research in Auditory Perception [mailto:AUDITORY@xxxxxxxx Im Auftrag von #ARIJIT BISWAS# Gesendet: Mittwoch, 21. August 2024 16:23 An: AUDITORY@xxxxxxxx <mailto:AUDITORY@xxxxxxxx>=20 Betreff: Re: Fwd: [AUDITORY] MUSHRA test with open reference; and then without open reference =20 ACHTUNG! Diese E-Mail kommt von Extern! WARNING! This email originated off-campus. Hi Hannes: =20 Thank you for bringing ITU-R BS.2132-0 to my attention=97I wasn't aware = of it! =20 I also appreciate you sharing the observations from your experiments. =20 Best regards, Arijit =20 =20 =20 _____ =20 From: AUDITORY - Research in Auditory Perception on behalf of Hannes Helmholz Sent: Tuesday, August 20, 2024 3:43 PM To: <mailto:AUDITORY@xxxxxxxx> AUDITORY@xxxxxxxx Subject: Re: Fwd: [AUDITORY] MUSHRA test with open reference; and then without open reference=20 =20 This is an interesting subject. There is a follow-up recommendation that could apply to your context. There is no catchy abbreviation in the recommendation, but its developers (Fraunhofer IDMT) endorse the term MuSCR (multi-stimulus category rating, in the sense of different categories, i.e., perceptual attributes, being evaluated for several conditions on separate MUSHRA-like pages/trials). ITU-R BS.2132-0, =93Method for the Subjective Quality Assessment of Audible Differences of Sound Systems using Multiple Stimuli without a Given Reference.=94 International Telecommunications Union, pp. 1=9618, = 2019. I'm unaware of studies directly comparing the results from MUSHRA and MuSCR. We exposed subjects to several conditions without a reference (although hidden high and low "anchor" conditions were included). They were asked to evaluate head-tracked binaural reproductions' "overall quality" from various microphone arrays. I consider this a relatively complex and open task since subjects must establish their internal reference on each trial/page of conditions. We observed trends similar to those in the existing literature that employed MUSHRA in comparable studies. However, there were more considerable variations and inconsistencies than we have seen in former MUSHRA experiments. I would say this is not surprising, considering the nature of the task. ITU-R BS.2132-0 recommends using expert listeners as a countermeasure to large variances, which we could not reasonably implement. An alternative idea would be employing more subjects and excluding listeners based on some quantifiable consistency measure (although this would have to be evaluated and justified very carefully). Best wishes, /Hannes PhD Student, Chalmers University of Technology, Gothenburg, Sweden On 2024-08-20 11:18, Raul Sanchez-Lopez wrote: > Dear Arijit, > > That's an excellent point. As of yet, I haven't come across any = studies > directly investigating that specific comparison. However, the > development of MUSHRA likely included such an evaluation. The method = is > defined in ITU-R Recommendation BS.1534-3. The introduction states: > > "This Recommendation describes a method for the subjective assessment = of > intermediate audio quality. This method reflects many aspects of > Recommendation ITU-R BS.1116 and incorporates the same grading scale > used for picture quality evaluation (i.e. Recommendation ITU-R = BT.500)." > > While BS.1116 features a hidden reference, it lacks a hidden anchor. = To > assess intermediate audio quality, a new method was developed (more > details here: = https://secure.aes.org/forum/pubs/conferences/?elib=3D8056). > > Crucially, analyzing how the panel utilizes the scale requires both = the > Reference and the Anchor for meaningful difference evaluations. If > there's no reference, it's simply not MUSHRA but a different type of > assessment. In MUSHRA, participants perform multi-comparisons and = place > their ratings between the reference and anchor. Without a reference, > they need to establish one first. This increases the risk of > inconsistency across repetitions, potentially leading to noisy data. > > Would you be willing to share some more details about your experiment? > I've dealt with similar questions in the past and may be able to offer > assistance or point you towards someone more experienced, like Force > Technology, for further guidance. > > Best wishes, > > --- > Raul Sanchez-Lopez > Hearing scientist | Audio Engineer | Technical Audiologist > Institute of Globally Distributed Open Research and Education > > On 2024-08-20 10:52, Raul Sanchez-Lopez wrote: >> ---------- Forwarded message --------- >> From: #ARIJIT BISWAS# < <mailto:000003292f44871c-dmarc-request@xxxxxxxx> 000003292f44871c-dmarc-request@xxxxxxxx> >> Date: Tue, 20 Aug 2024 at 09:14 >> Subject: [AUDITORY] MUSHRA test with open reference; and then without >> open reference >> To: < <mailto:AUDITORY@xxxxxxxx> AUDITORY@xxxxxxxx> >> >> Dear all: >> >> Is there any research on how subjective ratings in a MUSHRA test >> might be affected if the same systems (i.e., hidden reference, >> anchors, codecs under test) are re-evaluated without the ability to >> compare them against an open reference? >> >> If no such studies/papers exist, can we make any educated guesses >> about the potential outcomes? >> I imagine that, in the absence of a reference, the task for the >> subjects would become more difficult. >> Any other guesses? >> >> Thank you. >> >> Best regards, >> Arijit >> >> **Disclaimer** The sender of this email does not represent Nanyang >> Technological University and this email does not express the views or >> opinions of the University. **Disclaimer** The sender of this email does not represent Nanyang Technological University and this email does not express the views or opinions of the University.=20 ------=_NextPart_000_0093_01DAF533.0C80B630 Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable <html xmlns:v=3D"urn:schemas-microsoft-com:vml" = xmlns:o=3D"urn:schemas-microsoft-com:office:office" = xmlns:w=3D"urn:schemas-microsoft-com:office:word" = xmlns:m=3D"http://schemas.microsoft.com/office/2004/12/omml" = xmlns=3D"http://www.w3.org/TR/REC-html40"><head><meta = http-equiv=3DContent-Type content=3D"text/html; = charset=3Diso-8859-1"><meta name=3DGenerator content=3D"Microsoft Word = 15 (filtered medium)"><style></style></head><body lang=3DEN-CA link=3Dblue = vlink=3Dpurple style=3D'word-wrap:break-word'><div = class=3DWordSection1>Dear = all,<o:p></o:p><o:p> </o:p>= I’m interested to = hear the cohort’s opinion on the suitability of these parametric = tests based upon the need to transform the data, e.g. attempting to = adjust for rating variance.<o:p></o:p>Any question of reliability = is surely amplified by the unacceptably low number of participants in = many (most) listening experiments?<o:p></o:p><o:p> </o:p>= Mendon=E7a, C. and = Delikaris-Manias, S. (2018) ‘Statistical tests with MUSHRA = data’, in Audio engineering society convention 144. = Available at: <a = href=3D"http://www.aes.org/e-lib/browse.cfm?elib=3D19402">http://www.aes.= org/e-lib/browse.cfm?elib=3D19402</a> (Accessed: 23 August = 2024).<o:p></o:p><o:p> </o:p>= Thank you for the = discussion.<o:p></o:p><o:p> </o:p>= <div>______________________= _____________________<o:p></o:p>Jonathan J = Digby<o:p></o:p>Audio Engineer; = Digbyphonic Ltd<o:p></o:p>Postgraduate = Researcher; University of Derby, GBR<o:p></o:p>Tel: +44 7833 151 = 181<o:p></o:p></div><o:p> </o:p>= <div><div style=3D'border:none;border-top:solid #E1E1E1 = 1.0pt;padding:3.0pt 0cm 0cm 0cm'>From:<= /b> Thomas = Biberger <thomas.biberger@xxxxxxxx> Sent: = Thursday, August 22, 2024 7:08 AM Subject: AW: Fwd: [AUDITORY] = MUSHRA test with open reference; and then without open = reference<o:p></o:p></div></div><o:p> </o:p>Dear Arijit et al.,<o:p></o:p><o:p> </o:p>a study that goes in this direction was = recently carried out in Oldenburg by Henning Schepker, Florian Denk, = Birger Kollmeier and Simon Doclo. They investigated acoustic = transparency in hearables  and for this they made a perceptual = sound quality evaluation of commercially available hearables with 20 = normal-hearing listeners. In their 1st Experiment an explicit reference = was provided to the participants,  while in the 2nd Experiment no = explicit reference was provided. <o:p></o:p><o:p> </o:p>In their conclusions they state that "a = comparison of the discrimination and reliability of the quality ratings = of both experiments showed that participants are generally more reliable = and can discriminate better when a reference signal is = provided."<o:p></o:p><o:p> </o:p>For more details, = see<o:p></o:p><o:p> </o:p>H. Schepker, F. Denk, B. Kollmeier, and S. = Doclo, “Acoustic Transparency in Hearables–Perceptual Sound = Quality Evaluations”<o:p></o:p>J. Audio Eng. Soc., vol. 68, no. 7/8, pp. = 495–507, (2020 July/August.).<o:p></o:p>DOI: <a = href=3D"https://doi.org/10.17743/jaes.2020.0045">https://doi.org/10.17743= /jaes.2020.0045</a><o:p></o:p><o:p> </o:p>Hope that helps!<o:p></o:p><o:p> </o:p>BR, Thomas<o:p></o:p><o:p> </o:p>--<o:p></o:p><o:p> </o:p>Dr. Thomas Biberger<o:p></o:p><o:p> </o:p>Medizinische Physik and Cluster of = Excellence Hearing4all<o:p></o:p>Psychoacoustics & Auditory = Modelling<o:p></o:p>Universit=E4t = Oldenburg<o:p></o:p>D-26111 Oldenburg<o:p></o:p>Germany<o:p></o:p><o:p> </o:p>phone:+49-441-798 = 3528<o:p></o:p>room: W30-3-330<o:p></o:p>email: <a = href=3D"mailto:thomas.biberger@xxxxxxxx">thomas.biberger@xxxxxxxx= nburg.de</a><o:p></o:p><o:p> </o:p><div><div = style=3D'border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm = 0cm 0cm'>Von:</= b> AUDITORY - = Research in Auditory Perception [<a = href=3D"mailto:AUDITORY@xxxxxxxx">mailto:AUDITORY@xxxxxxxx<= /a>] Im Auftrag von #ARIJIT BISWAS# Gesendet: Mittwoch, = 21. August 2024 16:23 An: <a = href=3D"mailto:AUDITORY@xxxxxxxx">AUDITORY@xxxxxxxx</a> = Betreff: Re: Fwd: [AUDITORY] MUSHRA test with open reference; and = then without open reference<o:p></o:p></div></div><o:p> </o:p><div = style=3D'border:solid #003333 1.0pt;padding:2.0pt 2.0pt 2.0pt 2.0pt'>ACHTUNG! = Diese E-Mail kommt von Extern! WARNING! = This email originated off-campus.<o:p></o:p></div><div><div>H= i Hannes:<o:p></o:p></div><div><= o:p> </o:p></div><div>T= hank you for bringing ITU-R BS.2132-0 to my attention—I wasn't = aware of it!<o:p></o:p></div><div><= o:p> </o:p></div><div>I= also appreciate you sharing the observations from your = experiments.<o:p></o:p></div><div><= o:p> </o:p></div><div>B= est regards,<o:p></o:p></div><div>A= rijit<o:p></o:p></div><div><= o:p> </o:p></div><div id=3DSignature><o:= p> </o:p></div><div><o:p> </o:p><= /span></div><div class=3DMsoNormal align=3Dcenter = style=3D'text-align:center'><hr size=3D1 width=3D"98%" = align=3Dcenter></div>From:<s= pan lang=3DEN-US = style=3D'font-family:"Calibri",sans-serif;color:black'> AUDITORY - = Research in Auditory Perception on behalf of Hannes = Helmholz Sent: Tuesday, August 20, 2024 3:43 = PM To: <a = href=3D"mailto:AUDITORY@xxxxxxxx">AUDITORY@xxxxxxxx</a> Subject:</b= > Re: Fwd: [AUDITORY] MUSHRA test with open reference; and then = without open reference <o:p></o:p><div><o:p> </o:p><= /span></div><div>This is an interesting subject. There is a = follow-up recommendation that could apply to your context. There is = no catchy abbreviation in the recommendation, but its developers = (Fraunhofer IDMT) endorse the term MuSCR (multi-stimulus category = rating, in the sense of different categories, i.e.,  perceptual = attributes, being evaluated for several conditions on separate = MUSHRA-like pages/trials). ITU-R BS.2132-0, “Method for the = Subjective Quality Assessment of Audible Differences of Sound Systems = using Multiple Stimuli without a Given Reference.” = International Telecommunications Union, pp. 1–18, 2019. I'm = unaware of studies directly comparing the results from MUSHRA = and MuSCR. We exposed subjects to several conditions without a = reference (although hidden high and low "anchor" conditions = were included). They were asked to evaluate head-tracked binaural = reproductions' "overall quality" from various microphone = arrays. I consider this a relatively complex and open task since = subjects must establish their internal reference on each trial/page = of conditions. We observed trends similar to those in the existing = literature that employed MUSHRA in comparable studies. However, there = were more considerable variations and inconsistencies than we have = seen in former MUSHRA experiments. I would say this is not = surprising, considering the nature of the task. ITU-R = BS.2132-0 recommends using expert listeners as a countermeasure = to large variances, which we could not reasonably implement. An = alternative idea would be employing more subjects and excluding = listeners based on some quantifiable consistency measure (although = this would have to be evaluated and justified very = carefully). Best wishes, /Hannes PhD Student, Chalmers = University of Technology, Gothenburg, Sweden On 2024-08-20 11:18, = Raul Sanchez-Lopez wrote: > Dear Arijit, > > That's an = excellent point. As of yet, I haven't come across any studies > = directly investigating that specific comparison. However, the > = development of MUSHRA likely included such an evaluation. The method = is > defined in ITU-R Recommendation BS.1534-3. The introduction = states: > > "This Recommendation describes a method for = the subjective assessment of > intermediate audio quality. This = method reflects many aspects of > Recommendation ITU-R BS.1116 and = incorporates the same grading scale > used for picture quality = evaluation (i.e. Recommendation ITU-R BT.500)." > > = While BS.1116 features a hidden reference, it lacks a hidden anchor. = To > assess intermediate audio quality, a new method was developed = (more > details here: <a = href=3D"https://secure.aes.org/forum/pubs/conferences/?elib=3D8056)">http= s://secure.aes.org/forum/pubs/conferences/?elib=3D8056)</a>. > &= gt; Crucially, analyzing how the panel utilizes the scale requires both = the > Reference and the Anchor for meaningful difference = evaluations. If > there's no reference, it's simply not MUSHRA but = a different type of > assessment. In MUSHRA, participants perform = multi-comparisons and place > their ratings between the reference = and anchor. Without a reference, > they need to establish one = first. This increases the risk of > inconsistency across = repetitions, potentially leading to noisy data. > > Would = you be willing to share some more details about your experiment? > = I've dealt with similar questions in the past and may be able to = offer > assistance or point you towards someone more experienced, = like Force > Technology, for further guidance. > > Best = wishes, > > --- > Raul Sanchez-Lopez > Hearing = scientist | Audio Engineer | Technical Audiologist > Institute of = Globally Distributed Open Research and Education > > On = 2024-08-20 10:52, Raul Sanchez-Lopez wrote: >> ---------- = Forwarded message --------- >> From: #ARIJIT BISWAS# = <<a = href=3D"mailto:000003292f44871c-dmarc-request@xxxxxxxx">000003292f44871c-dmarc-request@xxxxxxxx</a></s= pan>> >> Date: = Tue, 20 Aug 2024 at 09:14 >> Subject: [AUDITORY] MUSHRA test = with open reference; and then without >> open = reference >> To: <<a = href=3D"mailto:AUDITORY@xxxxxxxx">AUDITORY@xxxxxxxx</a>> >> >> =  Dear all: >> >>  Is there any research on = how subjective ratings in a MUSHRA test >> might be affected if = the same systems (i.e., hidden reference, >> anchors, codecs = under test) are re-evaluated without the ability to >> compare = them against an open reference? >> >>  If no such = studies/papers exist, can we make any educated guesses >> about = the potential outcomes? >>  I imagine that, in the absence = of a reference, the task for the >> subjects would become more = difficult. >>  Any other guesses? >> >> =  Thank you. >> >>  Best regards, >> =  Arijit >> >>  **Disclaimer** The sender of = this email does not represent Nanyang >> Technological = University and this email does not express the views or >> = opinions of the University.<o:p></o:p></div>**Disclaimer** The sender of this = email does not represent Nanyang Technological University and this email = does not express the views or opinions of the University. = <o:p></o:p></div></div></body></html> ------=_NextPart_000_0093_01DAF533.0C80B630--

This message came from the mail archive
postings/2024/
maintained by:

DAn Ellis <dpwe@ee.columbia.edu>
Electrical Engineering Dept., Columbia University