Subject: Re: [AUDITORY] Fwd: [AUDITORY] MUSHRA test with open reference; and then without open reference From: #ARIJIT BISWAS# <000003292f44871c-dmarc-request@xxxxxxxx> Date: Tue, 20 Aug 2024 16:09:08 +0000--_000_SEZPR01MB4679BAD0710B5772645597CB908D2SEZPR01MB4679apcp_ Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable Hi Raul: Thanks for sharing your opinions. I don=92t have any planned subjective exp= eriments in mind. I was just curious about what might happen if there were no open reference. I appreciate your insight: "Without a reference, they need to establish one first. This increases the = risk of inconsistency across repetitions, potentially leading to noisy data= ." Best regards, Arijit ________________________________ From: Raul Sanchez-Lopez <raul.sanchezlopez@xxxxxxxx> Sent: Tuesday, August 20, 2024 11:18 AM To: #ARIJIT BISWAS# <PA9501460@xxxxxxxx> Cc: AUDITORY@xxxxxxxx <AUDITORY@xxxxxxxx> Subject: Re: Fwd: [AUDITORY] MUSHRA test with open reference; and then with= out open reference Dear Arijit, That's an excellent point. As of yet, I haven't come across any studies directly investigating that specific comparison. However, the development of MUSHRA likely included such an evaluation. The method is defined in ITU-R Recommendation BS.1534-3. The introduction states: "This Recommendation describes a method for the subjective assessment of intermediate audio quality. This method reflects many aspects of Recommendation ITU-R BS.1116 and incorporates the same grading scale used for picture quality evaluation (i.e. Recommendation ITU-R BT.500)." While BS.1116 features a hidden reference, it lacks a hidden anchor. To assess intermediate audio quality, a new method was developed (more details here: https://secure.aes.org/forum/pubs/conferences/?elib=3D8056). Crucially, analyzing how the panel utilizes the scale requires both the Reference and the Anchor for meaningful difference evaluations. If there's no reference, it's simply not MUSHRA but a different type of assessment. In MUSHRA, participants perform multi-comparisons and place their ratings between the reference and anchor. Without a reference, they need to establish one first. This increases the risk of inconsistency across repetitions, potentially leading to noisy data. Would you be willing to share some more details about your experiment? I've dealt with similar questions in the past and may be able to offer assistance or point you towards someone more experienced, like Force Technology, for further guidance. Best wishes, --- Raul Sanchez-Lopez Hearing scientist | Audio Engineer | Technical Audiologist Institute of Globally Distributed Open Research and Education On 2024-08-20 10:52, Raul Sanchez-Lopez wrote: > ---------- Forwarded message --------- > From: #ARIJIT BISWAS# <000003292f44871c-dmarc-request@xxxxxxxx> > Date: Tue, 20 Aug 2024 at 09:14 > Subject: [AUDITORY] MUSHRA test with open reference; and then without > open reference > To: <AUDITORY@xxxxxxxx> > > Dear all: > > Is there any research on how subjective ratings in a MUSHRA test > might be affected if the same systems (i.e., hidden reference, > anchors, codecs under test) are re-evaluated without the ability to > compare them against an open reference? > > If no such studies/papers exist, can we make any educated guesses > about the potential outcomes? > I imagine that, in the absence of a reference, the task for the > subjects would become more difficult. > Any other guesses? > > Thank you. > > Best regards, > Arijit > > **Disclaimer** The sender of this email does not represent Nanyang > Technological University and this email does not express the views or > opinions of the University. **Disclaimer** The sender of this email does not represent Nanyang Technolo= gical University and this email does not express the views or opinions of t= he University. --_000_SEZPR01MB4679BAD0710B5772645597CB908D2SEZPR01MB4679apcp_ Content-Type: text/html; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable <html> <head> <meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3DWindows-1= 252"> <style type=3D"text/css" style=3D"display:none;"> P {margin-top:0;margin-bo= ttom:0;} </style> </head> <body dir=3D"ltr"> <div class=3D"elementToProof" style=3D"font-family: Verdana, Geneva, sans-s= erif; font-size: 10pt; color: rgb(0, 0, 0);"> Hi Raul:</div> <div style=3D"font-family: Verdana, Geneva, sans-serif; font-size: 10pt; co= lor: rgb(0, 0, 0);"> <br> </div> <div class=3D"elementToProof" style=3D"font-family: Verdana, Geneva, sans-s= erif; font-size: 10pt; color: rgb(0, 0, 0);"> Thanks for sharing your opinions. I don=92t have any planned subjective exp= eriments in mind.</div> <div class=3D"elementToProof" style=3D"font-family: Verdana, Geneva, sans-s= erif; font-size: 10pt; color: rgb(0, 0, 0);"> I was just curious about what might happen if there were no open reference.= </div> <div class=3D"elementToProof" style=3D"font-family: Verdana, Geneva, sans-s= erif; font-size: 10pt; color: rgb(0, 0, 0);"> I appreciate your insight:</div> <div class=3D"elementToProof" style=3D"font-family: Verdana, Geneva, sans-s= erif; font-size: 10pt; color: rgb(0, 0, 0);"> <i>"Without a reference, they need to establish one first. This increa= ses the risk of inconsistency across repetitions, potentially leading to no= isy data."</i></div> <div style=3D"font-family: Verdana, Geneva, sans-serif; font-size: 10pt; co= lor: rgb(0, 0, 0);"> <br> </div> <div style=3D"font-family: Verdana, Geneva, sans-serif; font-size: 10pt; co= lor: rgb(0, 0, 0);"> Best regards,</div> <div class=3D"elementToProof" style=3D"font-family: Verdana, Geneva, sans-s= erif; font-size: 10pt; color: rgb(0, 0, 0);"> Arijit</div> <div class=3D"elementToProof" style=3D"font-family: Verdana, Geneva, sans-s= erif; font-size: 11pt; color: rgb(0, 0, 0);"> <br> </div> <div id=3D"Signature"> <div align=3D"left" style=3D"margin: 0cm 0cm 0pt; font-family: Verdana; fon= t-size: 7.5pt; color: gray;"> <br> </div> <div align=3D"left" style=3D"margin: 0cm 0cm 0pt;"></div> </div> <div id=3D"appendonsend"></div> <hr style=3D"display:inline-block;width:98%" tabindex=3D"-1"> <div id=3D"divRplyFwdMsg" dir=3D"ltr"><font face=3D"Calibri, sans-serif" st= yle=3D"font-size:11pt" color=3D"#000000"><b>From:</b> Raul Sanchez-Lopez &l= t;raul.sanchezlopez@xxxxxxxx><br> <b>Sent:</b> Tuesday, August 20, 2024 11:18 AM<br> <b>To:</b> #ARIJIT BISWAS# <PA9501460@xxxxxxxx><br> <b>Cc:</b> AUDITORY@xxxxxxxx <AUDITORY@xxxxxxxx><br> <b>Subject:</b> Re: Fwd: [AUDITORY] MUSHRA test with open reference; and th= en without open reference</font> <div> </div> </div> <div class=3D"BodyFragment"><font size=3D"2"><span style=3D"font-size:11pt;= "> <div class=3D"PlainText">Dear Arijit,<br> <br> That's an excellent point. As of yet, I haven't come across any studies <br= > directly investigating that specific comparison. However, the <br> development of MUSHRA likely included such an evaluation. The method is <br= > defined in ITU-R Recommendation BS.1534-3. The introduction states:<br> <br> "This Recommendation describes a method for the subjective assessment = of <br> intermediate audio quality. This method reflects many aspects of <br> Recommendation ITU-R BS.1116 and incorporates the same grading scale <br> used for picture quality evaluation (i.e. Recommendation ITU-R BT.500).&quo= t;<br> <br> While BS.1116 features a hidden reference, it lacks a hidden anchor. To <br= > assess intermediate audio quality, a new method was developed (more <br> details here: <a href=3D"https://secure.aes.org/forum/pubs/conferences/?eli= b=3D8056)"> https://secure.aes.org/forum/pubs/conferences/?elib=3D8056)</a>.<br> <br> Crucially, analyzing how the panel utilizes the scale requires both the <br= > Reference and the Anchor for meaningful difference evaluations. If <br> there's no reference, it's simply not MUSHRA but a different type of <br> assessment. In MUSHRA, participants perform multi-comparisons and place <br= > their ratings between the reference and anchor. Without a reference, <br> they need to establish one first. This increases the risk of <br> inconsistency across repetitions, potentially leading to noisy data.<br> <br> Would you be willing to share some more details about your experiment? <br> I've dealt with similar questions in the past and may be able to offer <br> assistance or point you towards someone more experienced, like Force <br> Technology, for further guidance.<br> <br> Best wishes,<br> <br> ---<br> Raul Sanchez-Lopez<br> Hearing scientist | Audio Engineer | Technical Audiologist<br> Institute of Globally Distributed Open Research and Education<br> <br> On 2024-08-20 10:52, Raul Sanchez-Lopez wrote:<br> > ---------- Forwarded message ---------<br> > From: #ARIJIT BISWAS# <000003292f44871c-dmarc-request@xxxxxxxx= ca><br> > Date: Tue, 20 Aug 2024 at 09:14<br> > Subject: [AUDITORY] MUSHRA test with open reference; and then without<= br> > open reference<br> > To: <AUDITORY@xxxxxxxx><br> > <br> > Dear all:<br> > <br> > Is there any research on how subjective ratings in a MUSHRA test= <br> > might be affected if the same systems (i.e., hidden reference,<br> > anchors, codecs under test) are re-evaluated without the ability to<br= > > compare them against an open reference?<br> > <br> > If no such studies/papers exist, can we make any educated guesse= s<br> > about the potential outcomes?<br> > I imagine that, in the absence of a reference, the task for the<= br> > subjects would become more difficult.<br> > Any other guesses?<br> > <br> > Thank you.<br> > <br> > Best regards,<br> > Arijit<br> > <br> > **Disclaimer** The sender of this email does not represent Nanya= ng<br> > Technological University and this email does not express the views or<= br> > opinions of the University.<br> </div> </span></font></div> **Disclaimer** The sender of this email does not represent Nanyang Technolo= gical University and this email does not express the views or opinions of t= he University. </body> </html> --_000_SEZPR01MB4679BAD0710B5772645597CB908D2SEZPR01MB4679apcp_--