Subject: Re: [AUDITORY] Registered reports From: Les Bernstein <lbernstein@xxxxxxxx> Date: Mon, 11 Jun 2018 10:55:01 -0400 List-Archive:<http://lists.mcgill.ca/scripts/wa.exe?LIST=AUDITORY>--------------848B8FC49368287683052A5C Content-Type: text/html; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-MIME-Autoconverted: from 8bit to quoted-printable by edgeum2.it.mcgill.ca id w5BEt52x005778 <html> <head> <meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Dutf= -8"> </head> <body text=3D"#000000" bgcolor=3D"#FFFFFF"> <font size=3D"-1"><font face=3D"Verdana">I agree with Ken and Roger.=C2= =A0 It's neither clear that the current system falls short nor that RRs would, effectively, solve any such problem.=C2=A0 To the degr= ee there is a problem, I fail to see how making RRs VOLUNTARY would serve as an effective remedy or, voluntary or not, serve to increase "standards of publication."=C2=A0 If people wish to have= the option, that sounds benign enough, save for the extra work required of reviewers.<br> <br> As suggested by Matt, I tried to think of the "wasted hours spent by investigators who repeat the failed methods of their peers and predecessors, only because the outcomes of failed experiments were never published."=C2=A0 Across the span of my career, for me and for those with whom I've worked, I can't identify that such wasted hours have been spent. As Ken notes, well-formed, well-motivated experiments employing sound methods should be (and are) published.<br> <br> Likewise, re Matt's comments, I cannot recall substantial instances of scientists "who cling to theories based on initial publications of work that later fails replication, but where those failed replications never get published."=C2=A0 Au contrair= e.=C2=A0 I can think of a quite a few cases in which essential replication failed, those findings were published, and the field was advanced.=C2=A0 I don't believe that it is the case that many= of us are clinging to theories that are invalid but for the publication of failed replications.=C2=A0 Theories gain status vi= a converging evidence.<br> <br> It seems to me that for what some are arguing would, essentially, be an auditory version of The Journal of Negative Results (<a class=3D"moz-txt-link-freetext" href=3D"https://en.wikipedia.org/wiki/Journal_of_Negative_Results_in_Biom= edicine">https://en.wikipedia.org/wiki/Journal_of_Negative_Results_in_Bio= medicine</a>).<br> <br> Still, if some investigators wish to have the RR option and journals are willing to offer it, then, by all means, have at it.=C2=A0 The proof of the pudding will be in the tasting.<br> <br> Les<br> </font></font><br> <br> <div class=3D"moz-cite-prefix">On 6/9/2018 5:13 AM, Roger Watt wrote:= <br> </div> <blockquote type=3D"cite" cite=3D"mid:17580_1528536950_5B1B9F76_17580_14_1_3da204f568b64a9e829f6fdc= e35748ee@xxxxxxxx"> <meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Du= tf-8"> <meta name=3D"Generator" content=3D"Microsoft Word 15 (filtered medium)"> <style><!-- /* Font Definitions */ @xxxxxxxx {font-family:"Cambria Math"; panose-1:2 4 5 3 5 4 6 3 2 4;} @xxxxxxxx {font-family:Calibri; panose-1:2 15 5 2 2 2 4 3 2 4;} /* Style Definitions */ p.MsoNormal, li.MsoNormal, div.MsoNormal {margin:0cm; margin-bottom:.0001pt; font-size:11.0pt; font-family:"Calibri",sans-serif;} a:link, span.MsoHyperlink {mso-style-priority:99; color:blue; text-decoration:underline;} a:visited, span.MsoHyperlinkFollowed {mso-style-priority:99; color:purple; text-decoration:underline;} p.msonormal0, li.msonormal0, div.msonormal0 {mso-style-name:msonormal; mso-margin-top-alt:auto; margin-right:0cm; mso-margin-bottom-alt:auto; margin-left:0cm; font-size:11.0pt; font-family:"Calibri",sans-serif;} span.EmailStyle18 {mso-style-type:personal-reply; font-family:"Calibri",sans-serif; color:windowtext;} .MsoChpDefault {mso-style-type:export-only; font-size:10.0pt;} @xxxxxxxx WordSection1 {size:612.0pt 792.0pt; margin:72.0pt 72.0pt 72.0pt 72.0pt;} div.WordSection1 {page:WordSection1;} --></style><!--[if gte mso 9]><xml> <o:shapedefaults v:ext=3D"edit" spidmax=3D"1026" /> </xml><![endif]--><!--[if gte mso 9]><xml> <o:shapelayout v:ext=3D"edit"> <o:idmap v:ext=3D"edit" data=3D"1" /> </o:shapelayout></xml><![endif]--> <div class=3D"WordSection1"> <p class=3D"MsoNormal"><span style=3D"mso-fareast-language:EN-US"= >3 points:<o:p></o:p></span></p> <p class=3D"MsoNormal"><span style=3D"mso-fareast-language:EN-US"= ><o:p>=C2=A0</o:p></span></p> <p class=3D"MsoNormal"><span style=3D"mso-fareast-language:EN-US"= >1. The issue of RR is tied up with the logic of null hypothesis testing. There are only two outcomes for null hypothesis testing: (i) a tentative conclusion that the null hypothesis should be regarded as inconsistent with the data and (ii) no conclusion about the null hypothesis can be reached from the data. Neither outcome refers to the alternative hypothesis, which is never tested. A nice idea in the literature is the counter-null. If I have a sample of 42 and an effect size of 0.2 (r-family), then my result is not significant: it is not inconsistent with a population effect size of 0. It is equally not inconsistent with the counter-null, a population effect size of ~0.4. It is less inconsistent with all population effect sizes in between the null and the counter-null. (NHST forces all these double negatives).<o:p><= /o:p></span></p> <p class=3D"MsoNormal"><span style=3D"mso-fareast-language:EN-US"= ><o:p>=C2=A0</o:p></span></p> <p class=3D"MsoNormal"><span style=3D"mso-fareast-language:EN-US"= >2. The current system of publish when p<0.05 is easy to game, hence all the so-called questionable practices. Any new system, like RR, will in due course become easy to game. By a long shot, the easiest (invalid) way to get an inflated effect size and an inappropriately small p is to test more participants than needed and keep only the =E2=80=9Cbest=E2=80= =9D ones. RR will not prevent that.<o:p></o:p></span></p> <p class=3D"MsoNormal"><span style=3D"mso-fareast-language:EN-US"= ><o:p>=C2=A0</o:p></span></p> <p class=3D"MsoNormal"><span style=3D"mso-fareast-language:EN-US"= >3. NHST assumes random sampling, which no-one achieves. The forms of sampling we use in reality are all possibly subject to issues of non-independence of participants which leads to Type I error rates (false positives) that are well above 5%. <o:p></o:p></span></p> <p class=3D"MsoNormal"><span style=3D"mso-fareast-language:EN-US"= ><o:p>=C2=A0</o:p></span></p> <p class=3D"MsoNormal"><span style=3D"mso-fareast-language:EN-US"= >None of this is to argue against RR, just to observe that it doesn=E2=80=99t resolve many of the current problems. Any cla= im that it does, is in itself a kind of Type I error and Type I errors are very difficult to eradicate once accepted.<o:p></o= :p></span></p> <p class=3D"MsoNormal"><span style=3D"mso-fareast-language:EN-US"= ><o:p>=C2=A0</o:p></span></p> <p class=3D"MsoNormal"><span style=3D"mso-fareast-language:EN-US"= >Roger Watt<o:p></o:p></span></p> <p class=3D"MsoNormal"><span style=3D"mso-fareast-language:EN-US"= >Professor of Psychology<o:p></o:p></span></p> <p class=3D"MsoNormal"><span style=3D"mso-fareast-language:EN-US"= >University of Stirling<o:p></o:p></span></p> <p class=3D"MsoNormal"><a name=3D"_MailEndCompose" moz-do-not-send=3D"true"><span style=3D"mso-fareast-language:EN-US"><o:p>=C2=A0</o:p></spa= n></a></p> <span style=3D"mso-bookmark:_MailEndCompose"></span> <div> <div style=3D"border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm"> <p class=3D"MsoNormal"><b><span lang=3D"EN-US">From:</span></= b><span lang=3D"EN-US"> AUDITORY - Research in Auditory Perceptio= n [<a class=3D"moz-txt-link-freetext" href=3D"mailto:AUDITO= RY@xxxxxxxx">mailto:AUDITORY@xxxxxxxx</a>] <b>On Behalf Of </b>Ken Grant<br> <b>Sent:</b> 09 June 2018 06:19<br> <b>To:</b> <a class=3D"moz-txt-link-abbreviated" href=3D"= mailto:AUDITORY@xxxxxxxx">AUDITORY@xxxxxxxx</a><br> <b>Subject:</b> Re: Registered reports<o:p></o:p></span><= /p> </div> </div> <p class=3D"MsoNormal"><o:p>=C2=A0</o:p></p> <p class=3D"MsoNormal">Why aren=E2=80=99t these =E2=80=9Cfailed=E2= =80=9D experiments published? What=E2=80=99s the definition of a failed experiment anyway.=C2=A0<o:p></o:p></p> <div> <p class=3D"MsoNormal"><o:p>=C2=A0</o:p></p> </div> <div> <p class=3D"MsoNormal" style=3D"margin-bottom:12.0pt">I think t= hat if the scientific question is well formed and well motivated AND the methods sound and appropriate for addressing the question, then whatever the result may be, this seems like a good experiment and one that should be published.=C2=A0<o:p><= /o:p></p> <div id=3D"AppleMailSignature"> <p class=3D"MsoNormal">Sent from my iPhone<o:p></o:p></p> <div> <p class=3D"MsoNormal"><span style=3D"font-size:13.0pt">Ken= W. Grant, PhD</span><o:p></o:p></p> </div> <div> <p class=3D"MsoNormal"><span style=3D"font-size:13.0pt">Chi= ef, Scientific and Clinical Studies</span><o:p></o:p></p> </div> <div> <p class=3D"MsoNormal"><span style=3D"font-size:13.0pt">Nat= ional Military Audiology and Speech-Pathology Center (NMASC)<= /span><o:p></o:p></p> </div> <div> <p class=3D"MsoNormal"><span style=3D"font-size:13.0pt">Wal= ter Reed National Military Medical Center</span><o:p></o:p>= </p> </div> <div> <div> <p class=3D"MsoNormal"><span style=3D"font-size:13.0pt">B= ethesda, MD 20889</span><o:p></o:p></p> </div> <div> <p class=3D"MsoNormal"><a href=3D"mailto:kenneth.w.grant.civ@xxxxxxxx" moz-do-not-send=3D"true">kenneth.w.grant.civ@xxxxxxxx= </a><o:p></o:p></p> </div> <div> <p class=3D"MsoNormal"><a href=3D"mailto:ken.w.grant@xxxxxxxx" moz-do-not-send=3D"true">ken.w.grant@xxxxxxxx</a><o:= p></o:p></p> </div> <div> <p class=3D"MsoNormal"><span style=3D"font-size:13.0pt">O= ffice: =C2=A0301-319-7043</span><o:p></o:p></p> </div> <div> <p class=3D"MsoNormal"><span style=3D"font-size:13.0pt">C= ell: =C2=A0301-919-2957</span><o:p></o:p></p> </div> <div> <div> <p class=3D"MsoNormal"><o:p>=C2=A0</o:p></p> <div> <p class=3D"MsoNormal"><o:p>=C2=A0</o:p></p> <div> <p class=3D"MsoNormal"><o:p>=C2=A0</o:p></p> </div> </div> </div> </div> </div> </div> <div> <p class=3D"MsoNormal" style=3D"margin-bottom:12.0pt"><br> On Jun 9, 2018, at 12:48 AM, Matthew Winn <<a href=3D"mailto:mwinn2@xxxxxxxx" moz-do-not-send=3D"true">mw= inn2@xxxxxxxx</a>> wrote:<o:p></o:p></p> </div> <blockquote style=3D"margin-top:5.0pt;margin-bottom:5.0pt"> <div> <div> <p class=3D"MsoNormal" style=3D"margin-bottom:12.0pt">The view that RRs will stifle progress is both true and false. While the increased load of advanced registration and rigidity in methods would, as Les points out, become burdensome for most of our basic work, there is another side to this. This is not a matter of morals (hiding a bad result, or fabricating a good result) or how to do our experiments. It=E2=80=99= s a matter of the standards of *publication*, which you will notice was the scope of Tim=E2=80=99s original cal= l to action. In general, we only ever read about experiments that came out well (and not the ones that didn=E2=80=99t). If there is a solution to that problem= , then we should consider it, or at least acknowledge that some solution might be needed. This is partly the culture of scientific journals, and partly the culture of the institutions that employ us. There's no need to question anybody's integrity in order to appreciate some benefit of RRs. <br> <br> Think for a moment about the amount of wasted hours spent by investigators who repeat the failed methods of their peers and predecessors, only because the outcomes of failed experiments were never published. Or those of us who cling to theories based on initial publications of work that later fails replication, but where those failed replications never get published. THIS stifles progress as well. If results were to be reported whether or not they come out as planned, we=E2= =80=99d have a much more complete picture of the evidence for and against the ideas. Julia's story also resonates with me; we've all reviewed papers where we've thought "if only the authors had sought input before running this labor-intensive study, the data would be so much more valuable."<br> <br> The arguments against RRs in this thread appear in my mind to be arguments against *compulsory* RRs for *all* papers in *all* journals, which takes the discussion off course. I have not heard such radical calls. If you don=E2=80=99t want to do a RR, then don=E2= =80=99t do it. But perhaps we can appreciate the goals of RR and see how those goals might be realized with practices that suit our own fields of work.<br> <br> Matt <o:p></o:p></p> <div> <div> <div> <div> <div> <div> <p class=3D"MsoNormal"><o:p>=C2=A0</o:p></p> </div> <div> <p class=3D"MsoNormal">----------------------= ----------------------------------------<o:p></o:p></p> </div> <div> <p class=3D"MsoNormal">Matthew Winn, Au.D., Ph.D.<br> Assistant Professor<br> Dept. of Speech & Hearing Sciences<br> University of Washington<o:p></o:p></p> </div> </div> </div> </div> </div> <p class=3D"MsoNormal"><o:p>=C2=A0</o:p></p> </div> </div> </div> </blockquote> </div> </div> <div align=3D"left"> <hr> <div align=3D"left"><font size=3D"2" face=3D"Arial">The Universit= y achieved an overall 5 stars in the QS World University Rankings 2018</font></div> <font size=3D"2" face=3D"Arial" color=3D"gray">The University of Stirling is a charity registered in Scotland, number SC 011159.<br> </font> </div> </blockquote> <br> <br> <div class=3D"moz-signature">-- <br> <meta http-equiv=3D"content-type" content=3D"text/html; charset=3Du= tf-8"> <title></title> <b><span style=3D"font-size:10.0pt;font-family:"Arial",sans-se= rif; mso-fareast-font-family:"Times New Roman \;color\:\#002048"">Leslie R. Bernstein, Ph.D. </span></b>= <b><span style=3D"font-size:10.0pt;font-family:"Arial",sans-serif; mso-fareast-font-family:"Times New Roman \;color\:\#585858"">| </span></b><span style=3D"font-size:10.0pt;font-family:"Arial",sans-serif;mso-fa= reast-font-family:"Times New Roman \;color\:\#585858"">Professor</span><span style=3D"mso-fareast-font-family: "Times New Roman""><o= :p></o:p></span><span style=3D"font-size:10.0pt;font-family:"Arial",sans-serif; mso-fareast-font-family:"Times New Roman \;color\:\#585858""><br> Depts. of Neuroscience and Surgery (Otolaryngology)| UConn School of Medicine </span><br> <span style=3D"font-size:10.0pt;font-family:"Arial",sans-seri= f; mso-fareast-font-family:"Times New Roman \;color\:\#585858""></span><span style=3D"font-size:10.0pt;font-family:"Arial",sans-seri= f; mso-fareast-font-family:"Times New Roman \;color\:\#585858"">263 Farmington Avenue, Farmington, CT 06030-3401</span><br> <span style=3D"font-size:10.0pt;font-family:"Arial",sans-seri= f; mso-fareast-font-family:"Times New Roman \;color\:\#585858""></span><span style=3D"font-size:10.0pt;font-family:"Arial",sans-seri= f; mso-fareast-font-family:"Times New Roman \;color\:\#585858"">Office: 860.679.4622 | Fax: 860.679.2495<br> <br> <img alt=3D"" src=3D"cid:part6.4DF1ADC2.B162E604@xxxxxxxx" height=3D"48" width=3D"125"><br> </span> </div> </body> </html> --------------848B8FC49368287683052A5C Content-Type: image/png; name="uconnhealth_stacked_blue_email.png" Content-ID: <part6.4DF1ADC2.B162E604@xxxxxxxx> Content-Disposition: inline; filename="uconnhealth_stacked_blue_email.png" Content-Transfer-Encoding: base64 iVBORw0KGgoAAAANSUhEUgAAAH0AAAAwCAMAAAALmIWlAAAAGXRFWHRTb2Z0d2FyZQBBZG9i ZSBJbWFnZVJlYWR5ccllPAAAADNQTFRFKzVYHCZM4eLn8PHzpKi30dTbaG+IlZqrs7fDWWB8 O0Nkd32UwsXPSlJwhoufDRhA////A68jmAAAABF0Uk5T/////////////////////wAlrZli AAACYklEQVR42uzY22KDIAwAUC7e6gjw/187gUC4OetaupflaTXKWSuBKNMuJptC+QMCP20a uPHBQCubBWU46K24uB7NfZ58ZqaMP8D8AEBHtT/goXnB8WPwBf8r1WYUXWxUOZobHcJXIMcf ONclmE4s0lrRzYBIOjvTjX5WV6YbP2ZQJ6TWjfiAbuYzHW7p+xpuolQrP83sjQ5nupnu6Lqa J92MbnSznulcfkBHpNXNY6y+c0JKnWX1eFeflQt5qcNESKnrQMrf6LSAXeiWkEoXdOU4nZBK xzPnoTohtW5ZZyK8W09Io6tYjyP1iDS6XbAeh815S0irS6zHUfXuB0Ok1S3W41AdEWh1/Nb7 UD1bWytd0LY4ThdnerYbBJ1taT5XBmy4J8mNnWagp5fbba6HeiTdneK6vNgoFj0EhyO4eSZT 6AlpdNXoNzqYi94GbIU0eqhHp3d5drRf4heZUo9Iq8vYptup6VB37H6m/W6m0hFpdaxH3/yK 9cHpVq5Zy59n2GMV/QxPGdV9NElPE9njyOozsfWWKq6j7bLuw97KPBnM/mX86//6n+tH/yLK P7AUXcyd80Lx53WnspB47ty5qKND1gLockfIFzHR7srpQ7X0+6GgHPNFXedPyaN1CPFli71z +ZBeTxbcI2RXxwOQj/lW/Ui610Z6gA4a32WRrssXYbO76V/Zy6FLnZVj/qCnqGcdZE1DeFKc ntWrMV/RZXCh7B/fpF/+8tQQx6J74y9/Oeuyrm75+Jyf8oKWH9CL1SYNJtJ91HTOAD2fdjPN 9UcsumwmDNYXqvPtOLa9pH8LMACnoV0siZAyOAAAAABJRU5ErkJggg== --------------848B8FC49368287683052A5C--