Subject: Re: [AUDITORY] Registered reports From: Les Bernstein <lbernstein@xxxxxxxx> Date: Thu, 7 Jun 2018 09:40:55 -0400 List-Archive:<http://lists.mcgill.ca/scripts/wa.exe?LIST=AUDITORY>--------------6631115EF2C0D14AA5683BBB Content-Type: text/html; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-MIME-Autoconverted: from 8bit to quoted-printable by edgeum2.it.mcgill.ca id w57DfRBj031698 <html> <head> <meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Dutf= -8"> </head> <body text=3D"#000000" bgcolor=3D"#FFFFFF"> <font size=3D"-1"><font face=3D"Verdana">Peter--<br> <br> I think you have benefited the discussion by focusing on the "confirmatory" vs. "exploratory" CONTINUUM.=C2=A0 Thank you!=C2=A0= I agree with these two (among others) of your statements:<br> </font></font><br> <font size=3D"-1"><font face=3D"Verdana">1) I think that the debate about registered reports deteriorates too easily into an all-or-nothing argument.<br> </font></font><font size=3D"-1"><font face=3D"Verdana">2) What we n= eed is honesty about where our research falls on this continuum.<br> <br> Despite your recognition of a continuum, your reply references a dichotomous view, given your reference to Tukey.=C2=A0 Consider t= his definition of "confirmatory research" from <a class=3D"moz-txt-link-freetext" href=3D"http://www.butlerscientifics.c= om/single-post/2014/10/08/Exploratory-vs-Confirmatory-Research">http://ww= w.butlerscientifics.com/single-post/2014/10/08/Exploratory-vs-Confirmator= y-Research</a>:<br> <br> </font></font><font size=3D"-1"><font face=3D"Verdana"><span style=3D"margin: 0px; padding: 0px; border: 0px; outline: 0px; vertical-align: baseline; background: 0px 0px; color: rgb(89, 89, 89); font-family: Basic, sans-serif; font-size: 18px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; letter-spacing: normal; orphans: 2; text-align: left; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial; font-weight: bold;">Co= nfirmatory research</span><span style=3D"color: rgb(89, 89, 89); font-family: Basic, sans-serif; font-size: 18px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: left; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;"><span>=C2=A0</span>(a.k.a.<span>=C2=A0= </span></span><span style=3D"margin: 0px; padding: 0px; border: 0px; outline: 0px; vertical-align: baseline; background: 0px 0px; color: rgb(89, 89, 89); font-family: Basic, sans-serif; font-size: 18px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: left; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: underline;"><a href=3D"https://explorable.com/statistical-hypothesis-testing= " target=3D"_blank" data-content=3D"https://explorable.com/statistical-hypothesis= -testing" data-type=3D"external" rel=3D"undefined" style=3D"margin: 0px= ; padding: 0px; border: 0px; outline: 0px; vertical-align: baseline; background: 0px 0px; cursor: pointer; text-decoration: none; color: inherit;">hypothesis testing</a= ></span><span style=3D"color: rgb(89, 89, 89); font-family: Basic, sans-serif= ; font-size: 18px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: left; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;">) is where researchers have a pretty good idea of what's going on. That is, researcher has a theory (or several theories), and the objective is to find out if the theory is supported by the facts.<br> <br> </span></font></font><font size=3D"-1"><font face=3D"Verdana"><sp= an style=3D"font-family: Basic,sans-serif; font-size: 18px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: left; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial; display: inline ! important; float: none;"><font size=3D"-1"><font face=3D"Verdan= a">While the notion of a continuum is helpful, I would argue that only a very narrow set of studies are ever merely "confirmatory."=C2=A0 The problem lies with the identificat= ion of "hypothesis testing" with "confirmatory."=C2=A0 As I see= it, it is a false equivalence and is naive.=C2=A0 In many cases= a study that seeks to test a hypothesis or hypotheses falls on your proposed continuum such that a registered reports (RR) would be undesirable for many of the reasons I identified in previous responses.=C2=A0 As I see it, the se= t of studies that might be appropriate for RRs are those that offer a virtually unequivocal, binary set of potential outcomes.=C2=A0 Those, in my view, are few and far between = and are, more often than not, relatively uninteresting.=C2=A0 S= o, if people wish to use RRs for such studies, then fine.=C2=A0 For the remainder-- most of scientific output-- RRs hold little value and could, in my view, serve to stifle progress.<br> <br> Les<br> </font></font></span></font></font> <div class=3D"moz-cite-prefix"><br> <br> On 6/6/2018 8:04 AM, Peter Harrison wrote:<br> </div> <blockquote type=3D"cite" cite=3D"mid:12041_1528344483_5B18AFA3_12041_417_19_5F3F7AC9-C5C2-4255-A1C= F-457E3A68781D@xxxxxxxx"> <meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Du= tf-8"> <div class=3D"">Dear list,</div> <div class=3D""><br class=3D""> </div> <div class=3D"">I=E2=80=99ve found this debate very interesting, th= ank you. Here are some thoughts of my own:</div> <div class=3D""><br class=3D""> </div> <div class=3D"">I think that the debate about registered reports deteriorates too easily into an all-or-nothing argument. Registered reports are ideal for confirmatory research, where it is realistic to specify the analysis in advance, where the hypothetico-deductive method makes sense, and where the researcher has a good plan of what the final paper should look like before they conduct the study. However, they are often not well-suited to exploratory research where the goal is simply to find out more about a given phenomenon. As noted already on this thread, in such cases the branching factor of potential analyses may be simply too high to be worth specifying in advance. This is particularly true when the researcher wishes to conduct follow-up experiments based on the results of previous experiments.</div> <div class=3D""><br class=3D""> </div> Confirmatory and exploratory research are both vital - neither one is sufficient by=C2=A0itself (see e.g. Turkey 1980=C2=A0-=C2=A0<a href=3D"https://urldefense.proofpoint.com/v2/url?u=3Dhttp-3A__www.jstor.o= rg_stable_2682991&d=3DDwMGaQ&c=3DEZxp_D7cDnouwj5YEFHgXuSKoUq2zVQZ= _7Fw9yfotck&r=3D2Pw2GwelGcMR4953G-STHGpPJm2-pYYYSPmTwJk3sWM&m=3DQ= gj_3ZpUHv4jHVkW8H2ur5onMjbdO0aRXlJg2eqSzb4&s=3DtIRQcUn_AXQVySU6FSm3c0= ihAcsI-voUUoP9ji6-mWE&e=3D" class=3D"" moz-do-not-send=3D"true">http://www.jstor.org/stable/2= 682991</a>). What we need is honesty about where our research falls on this continuum. Unfortunately the emphasis on the hypothetico-deductive method and null hypothesis significance testing in psychological teaching provides many with the impression that confirmatory studies with error control are the only =E2=80=98scientific=E2=80=99= way to generate knowledge. Perhaps if Bayesian techniques were more commonly taught then we=E2=80=99d be more likely to see the gradual accumulation of positive evidence as a valid alternative. <div class=3D""><br class=3D""> </div> <div class=3D"">On registered reports, then, I believe the following:</div> <div class=3D""><br class=3D""> </div> <div class=3D"">- they should be an good tool for preventing exploratory research from being published as confirmatory research=C2=A0</div> <div class=3D"">-> we should encourage journals to offer registered reports as an option</div> <div class=3D"">-> we should support Tim Schoof in the initiativ= e to write to hearing journals</div> <div class=3D"">-> if you conduct a confirmatory study, then submitting it as a registered report should be a good way to enhance the credibility of your findings</div> <div class=3D""><br class=3D""> </div> <div class=3D"">- registered reports are often not suitable for exploratory research=C2=A0</div> <div class=3D"">-> we shouldn=E2=80=99t let the impression persi= st that registered reports should be compulsory for all research.</div> <div class=3D""><br class=3D""> </div> <div class=3D"">Best wishes,</div> <div class=3D"">Peter<br class=3D""> <br class=3D""> <blockquote type=3D"cite" class=3D"">On 6 Jun 2018, at 09:57, Nil= esh Madhu <000000405df1884c-dmarc-<a href=3D"mailto:request@xxxxxxxx" class=3D"" moz-do-not-send=3D"true">request@xxxxxxxx</a>> wrote:<br class=3D""> <br class=3D""> Good morning Bas,<br class=3D""> <br class=3D""> I see your point and I do agree that, in cases such as those you mention, pre-defining and getting feedback on the trial procedure you wish to conduct makes=C2=A0sense. However, for su= ch scenarios, shouldn't there be industry-wide standards on=C2=A0testing/trials and reporting in place already?<br class= =3D""> <br class=3D""> If someone intentionally conducts a poorly conceived trial, the paper would/should=C2=A0be rejected in the peer-review anyw= ay ("reject/do not encourage resubmission").=C2=A0As I see it, registered reports can be helpful if researchers are not sure what trials=C2=A0they should conduct and want early feedback. E= ven here, perhaps, a simpler=C2=A0solution would be to make guideli= nes available, instead of enforcing an=C2=A0'administrative' layer.= <br class=3D""> <br class=3D""> Of course, all of this is assuming that the intent is to make registered reports=C2=A0compulsory for any/every article. If th= is is not the case, the point is moot :)<br class=3D""> <br class=3D""> Greetings from lovely, sunny Belgium (yes, we do have such days!)<br class=3D""> <br class=3D""> Nilesh<br class=3D""> <br class=3D""> PS: Massimo, I like your point about the carpenter being adaptive on the job.=C2=A0Previously I attributed this necessit= y to adapt to my poor skills ;)<br class=3D""> <br class=3D""> <br class=3D""> <br class=3D""> On 2018-06-06 09:35, Bas Van Dijk wrote:<br class=3D""> <blockquote type=3D"cite" class=3D"">Hi Nilesh,<br class=3D""> I agree to certain extend but I do feel that registered reporting<br class=3D""> makes sense for 'close to product' trials and trials that lead to<br class=3D""> treatments (for example evaluation of a fitting algorithm). In fact,<br class=3D""> it should not really be =C2=A0' double work' as you fear beca= use if you<br class=3D""> execute a poor trial and then try to get it published (believe me.. it<br class=3D""> happens :-) )but it gets rejected and you basically have no option but<br class=3D""> to redo (part of) the work. (and -re-writing the text to get a poor<br class=3D""> trial accepted for publication is of course exactly what you don't<br class=3D""> want...). =C2=A0That's more double work that writing up a goo= d trial<br class=3D""> proposal, have it reviewed and then know that if you execute according<br class=3D""> to plan it's likely to get published even if the results are negative<br class=3D""> or non-conclusive, that could be a pre as well.<br class=3D""= > Best wishes,<br class=3D""> Bas<br class=3D""> Bas Van Dijk<br class=3D""> Program Manager, A&A - Clinician and Research Tools<br class=3D""> Cochlear Technology Centre Belgium<br class=3D""> Schali=C3=ABnhoevedreef 20 I<br class=3D""> 2800 Mechelen<br class=3D""> BELGIUM<br class=3D""> Phone: +3215795528<br class=3D""> Mobile: +32473976270<br class=3D""> <a href=3D"mailto:BVanDijk@xxxxxxxx" class=3D"" moz-do-not-send=3D"true">Email: BVanDijk@xxxxxxxx</a><b= r class=3D""> <a class=3D"moz-txt-link-abbreviated" href=3D"http://www.coch= lear.com">www.cochlear.com</a><br class=3D""> -----Original Message-----<br class=3D""> From: AUDITORY - Research in Auditory Perception<br class=3D"= "> [<a class=3D"moz-txt-link-freetext" href=3D"mailto:AUDITORY@xxxxxxxx= ISTS.MCGILL.CA">mailto:AUDITORY@xxxxxxxx</a>] On Behalf Of Nilesh = Madhu<br class=3D""> Sent: dinsdag 5 juni 2018 13:16<br class=3D""> To: <a class=3D"moz-txt-link-abbreviated" href=3D"mailto:AUDI= TORY@xxxxxxxx">AUDITORY@xxxxxxxx</a><br class=3D""> Subject: [AUDITORY] Registered reports<br class=3D""> Dear Tim,<br class=3D""> I appreciate your initiative towards reproducible research. However I<br class=3D""> fear that registered reports would just add another layer of overhead<br class=3D""> to academics and students already under the pressure to publish. If I<br class=3D""> understand correctly, this involves two rounds of review: a first<br class=3D""> review based on the methodology and evaluation and a second based on<br class=3D""> the results of the research. For each stage, probably at least two<br class=3D""> review rounds would be needed (going by the current publishing cycle).<br class=3D""> I fear, as Gaston does, this might stifle creativity and lead to<br class=3D""> overwork also for reviewers and editors. Of course, this is assuming<br class=3D""> you want to make registered reports compulsory...<br class=3D""> Furthermore, such an approach may not be equally applicable to all<br class=3D""> research. For research into algorithms, for example, the value of the<br class=3D""> research lies, usually, in the core idea. There are myriad accepted<br class=3D""> forms of evaluation and to force a strict evaluation<br class=3D""> pattern/methodology would be counterproductive. Reproducible research<br class=3D""> in this case is targeted by encouraging authors to make their code and<br class=3D""> test data public.<br class=3D""> What I would support are (voluntary) guidelines on reporting results<br class=3D""> of experiments. This is often to be found in in the engineering field,<br class=3D""> when one participates in an open challenge.<br class=3D""> Lastly, the main reason for this initiative is to avoid 'mis-reporting'<br class=3D""> the results in favour of a hypothesis. Surely, this calls for self<br class=3D""> policing? Aren't we, as researchers, possessed of sufficient integrity<br class=3D""> and ethics to present our research in the correct light? If this core<br class=3D""> value is missing, I fear no external policing is going to help.<br class=3D""> Best regards<br class=3D""> Nilesh Madhu<br class=3D""> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= <br class=3D""> "The information contained in this e-mail message may be confidential<br class=3D""> information, and may also be privileged. If you are not the intended<br class=3D""> recipient, any use, interference with, disclosure or copying of this<br class=3D""> material is unauthorised and prohibited. If you have received this<br class=3D""> message in error, please notify us by return email and delete the<br class=3D""> original message."<br class=3D""> </blockquote> </blockquote> <br class=3D""> </div> </blockquote> <br> <br> <div class=3D"moz-signature">-- <br> <meta http-equiv=3D"content-type" content=3D"text/html; charset=3Du= tf-8"> <title></title> <b><span style=3D"font-size:10.0pt;font-family:"Arial",sans-se= rif; mso-fareast-font-family:"Times New Roman \;color\:\#002048"">Leslie R. Bernstein, Ph.D. </span></b>= <b><span style=3D"font-size:10.0pt;font-family:"Arial",sans-serif; mso-fareast-font-family:"Times New Roman \;color\:\#585858"">| </span></b><span style=3D"font-size:10.0pt;font-family:"Arial",sans-serif;mso-fa= reast-font-family:"Times New Roman \;color\:\#585858"">Professor</span><span style=3D"mso-fareast-font-family: "Times New Roman""><o= :p></o:p></span><span style=3D"font-size:10.0pt;font-family:"Arial",sans-serif; mso-fareast-font-family:"Times New Roman \;color\:\#585858""><br> Depts. of Neuroscience and Surgery (Otolaryngology)| UConn School of Medicine </span><br> <span style=3D"font-size:10.0pt;font-family:"Arial",sans-seri= f; mso-fareast-font-family:"Times New Roman \;color\:\#585858""></span><span style=3D"font-size:10.0pt;font-family:"Arial",sans-seri= f; mso-fareast-font-family:"Times New Roman \;color\:\#585858"">263 Farmington Avenue, Farmington, CT 06030-3401</span><br> <span style=3D"font-size:10.0pt;font-family:"Arial",sans-seri= f; mso-fareast-font-family:"Times New Roman \;color\:\#585858""></span><span style=3D"font-size:10.0pt;font-family:"Arial",sans-seri= f; mso-fareast-font-family:"Times New Roman \;color\:\#585858"">Office: 860.679.4622 | Fax: 860.679.2495<br> <br> <img alt=3D"" src=3D"cid:part5.17ABD5DC.AA243958@xxxxxxxx" height=3D"48" width=3D"125"><br> </span> </div> </body> </html> --------------6631115EF2C0D14AA5683BBB Content-Type: image/png; name="uconnhealth_stacked_blue_email.png" Content-ID: <part5.17ABD5DC.AA243958@xxxxxxxx> Content-Disposition: inline; filename="uconnhealth_stacked_blue_email.png" Content-Transfer-Encoding: base64 iVBORw0KGgoAAAANSUhEUgAAAH0AAAAwCAMAAAALmIWlAAAAGXRFWHRTb2Z0d2FyZQBBZG9i ZSBJbWFnZVJlYWR5ccllPAAAADNQTFRFKzVYHCZM4eLn8PHzpKi30dTbaG+IlZqrs7fDWWB8 O0Nkd32UwsXPSlJwhoufDRhA////A68jmAAAABF0Uk5T/////////////////////wAlrZli AAACYklEQVR42uzY22KDIAwAUC7e6gjw/187gUC4OetaupflaTXKWSuBKNMuJptC+QMCP20a uPHBQCubBWU46K24uB7NfZ58ZqaMP8D8AEBHtT/goXnB8WPwBf8r1WYUXWxUOZobHcJXIMcf ONclmE4s0lrRzYBIOjvTjX5WV6YbP2ZQJ6TWjfiAbuYzHW7p+xpuolQrP83sjQ5nupnu6Lqa J92MbnSznulcfkBHpNXNY6y+c0JKnWX1eFeflQt5qcNESKnrQMrf6LSAXeiWkEoXdOU4nZBK xzPnoTohtW5ZZyK8W09Io6tYjyP1iDS6XbAeh815S0irS6zHUfXuB0Ok1S3W41AdEWh1/Nb7 UD1bWytd0LY4ThdnerYbBJ1taT5XBmy4J8mNnWagp5fbba6HeiTdneK6vNgoFj0EhyO4eSZT 6AlpdNXoNzqYi94GbIU0eqhHp3d5drRf4heZUo9Iq8vYptup6VB37H6m/W6m0hFpdaxH3/yK 9cHpVq5Zy59n2GMV/QxPGdV9NElPE9njyOozsfWWKq6j7bLuw97KPBnM/mX86//6n+tH/yLK P7AUXcyd80Lx53WnspB47ty5qKND1gLockfIFzHR7srpQ7X0+6GgHPNFXedPyaN1CPFli71z +ZBeTxbcI2RXxwOQj/lW/Ui610Z6gA4a32WRrssXYbO76V/Zy6FLnZVj/qCnqGcdZE1DeFKc ntWrMV/RZXCh7B/fpF/+8tQQx6J74y9/Oeuyrm75+Jyf8oKWH9CL1SYNJtJ91HTOAD2fdjPN 9UcsumwmDNYXqvPtOLa9pH8LMACnoV0siZAyOAAAAABJRU5ErkJggg== --------------6631115EF2C0D14AA5683BBB--