Subject: Re: [AUDITORY] Registered reports From: Julia Strand <00000071c2dbe20f-dmarc-request@xxxxxxxx> Date: Fri, 8 Jun 2018 14:18:19 -0500 List-Archive:<http://lists.mcgill.ca/scripts/wa.exe?LIST=AUDITORY>--0000000000000f889f056e2645c3 Content-Type: multipart/alternative; boundary="0000000000000f889e056e2645c2" --0000000000000f889e056e2645c2 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi all, I'll add one thing to the discussion that hasn't been covered. I recently had my first Stage 1 registered report accepted (in principle) and we've just begun data collection for it. When we went through the first round of peer review, the reviewers raised several very interesting and important points we hadn't considered. We changed components of the experiment design to reflect their input and more cleanly test the question we set out to test, and the experiment is much stronger as a result. Had we gone through the process the traditional way, it is entirely possible that the original design would have been seen as flawed enough not to be publishable (or would require additional experimentation). I know some people are concerned that the RR process would result in extra work, but this is a case where getting feedback earlier in the process certainly reduced the amount of work and made the work better than it otherwise would have been. Best, Julia On Fri, Jun 8, 2018 at 2:41 AM, Schoof, Tim <t.schoof@xxxxxxxx> wrote: > Dear List, > > > It has been really great seeing such a lively discussion on the topic! > > > While it is clear that not everyone sees the benefit of registered report= s > - and they're probably not perfect, but hopefully they're a step in the > right direction - I also know that a lot of people do value them and woul= d > be interested in submitting their work as a registered report. I think it > would be good to at least have the option. My impression from psychology > journals that already offer registered reports is that they are not > compulsory. > > > Just over 30 people have contacted me these past few days to say they > would like to sign the letter > <https://osf.io/3wct2/wiki/Journal%20Requests/>. I plan to send this out > early next week. If anyone still wants to add their name to the list, jus= t > let me know. > > As a result of this discussion on the auditory list, I know that a few > editors / journals have started thinking and talking about the possibilit= y > of offering registered reports. I'd be more than happy to keep the list > updated on any decisions that are made by the various journals in the > coming months. > > > Best, > Tim > > > > ------------------------------ > *From:* AUDITORY - Research in Auditory Perception < > AUDITORY@xxxxxxxx> on behalf of Les Bernstein <lbernstein@xxxxxxxx= > > *Sent:* 07 June 2018 08:40 > *To:* AUDITORY@xxxxxxxx > *Subject:* {SPAM?} Re: Registered reports > > Peter-- > > I think you have benefited the discussion by focusing on the > "confirmatory" vs. "exploratory" CONTINUUM. Thank you! I agree with the= se > two (among others) of your statements: > > 1) I think that the debate about registered reports deteriorates too > easily into an all-or-nothing argument. > 2) What we need is honesty about where our research falls on this > continuum. > > Despite your recognition of a continuum, your reply references a > dichotomous view, given your reference to Tukey. Consider this definitio= n > of "confirmatory research" from http://www.butlerscientifics. > com/single-post/2014/10/08/Exploratory-vs-Confirmatory-Research: > > <http://www.butlerscientifics.com/single-post/2014/10/08/Exploratory-vs-C= onfirmatory-Research> > AutoDiscovery Automatic Intelligent Exploratory Data ... > <http://www.butlerscientifics.com/single-post/2014/10/08/Exploratory-vs-C= onfirmatory-Research> > www.butlerscientifics.com > AutoDiscovery is an automatic intelligent exploratory data analysis > software that unveils complex relationships in the data files of your > experiments. > > > Confirmatory research (a.k.a. hypothesis testing > <https://explorable.com/statistical-hypothesis-testing>) is where > researchers have a pretty good idea of what's going on. That is, research= er > has a theory (or several theories), and the objective is to find out if t= he > theory is supported by the facts. > > While the notion of a continuum is helpful, I would argue that only a ver= y > narrow set of studies are ever merely "confirmatory." The problem lies > with the identification of "hypothesis testing" with "confirmatory." As = I > see it, it is a false equivalence and is naive. In many cases a study th= at > seeks to test a hypothesis or hypotheses falls on your proposed continuum > such that a registered reports (RR) would be undesirable for many of the > reasons I identified in previous responses. As I see it, the set of > studies that might be appropriate for RRs are those that offer a virtuall= y > unequivocal, binary set of potential outcomes. Those, in my view, are fe= w > and far between and are, more often than not, relatively uninteresting. > So, if people wish to use RRs for such studies, then fine. For the > remainder-- most of scientific output-- RRs hold little value and could, = in > my view, serve to stifle progress. > > Les > > > On 6/6/2018 8:04 AM, Peter Harrison wrote: > > Dear list, > > I=E2=80=99ve found this debate very interesting, thank you. Here are some= thoughts > of my own: > > I think that the debate about registered reports deteriorates too easily > into an all-or-nothing argument. Registered reports are ideal for > confirmatory research, where it is realistic to specify the analysis in > advance, where the hypothetico-deductive method makes sense, and where th= e > researcher has a good plan of what the final paper should look like befor= e > they conduct the study. However, they are often not well-suited to > exploratory research where the goal is simply to find out more about a > given phenomenon. As noted already on this thread, in such cases the > branching factor of potential analyses may be simply too high to be worth > specifying in advance. This is particularly true when the researcher wish= es > to conduct follow-up experiments based on the results of previous > experiments. > > Confirmatory and exploratory research are both vital - neither one is > sufficient by itself (see e.g. Turkey 1980 - http://www.jstor.org/ > stable/2682991 > <https://urldefense.proofpoint.com/v2/url?u=3Dhttp-3A__www.jstor.org_stab= le_2682991&d=3DDwMGaQ&c=3DEZxp_D7cDnouwj5YEFHgXuSKoUq2zVQZ_7Fw9yfotck&r=3D2= Pw2GwelGcMR4953G-STHGpPJm2-pYYYSPmTwJk3sWM&m=3DQgj_3ZpUHv4jHVkW8H2ur5onMjbd= O0aRXlJg2eqSzb4&s=3DtIRQcUn_AXQVySU6FSm3c0ihAcsI-voUUoP9ji6-mWE&e=3D>). > What we need is honesty about where our research falls on this continuum. > Unfortunately the emphasis on the hypothetico-deductive method and null > hypothesis significance testing in psychological teaching provides many > with the impression that confirmatory studies with error control are the > only =E2=80=98scientific=E2=80=99 way to generate knowledge. Perhaps if B= ayesian techniques > were more commonly taught then we=E2=80=99d be more likely to see the gra= dual > accumulation of positive evidence as a valid alternative. > > On registered reports, then, I believe the following: > > - they should be an good tool for preventing exploratory research from > being published as confirmatory research > -> we should encourage journals to offer registered reports as an option > -> we should support Tim Schoof in the initiative to write to hearing > journals > -> if you conduct a confirmatory study, then submitting it as a registere= d > report should be a good way to enhance the credibility of your findings > > - registered reports are often not suitable for exploratory research > -> we shouldn=E2=80=99t let the impression persist that registered report= s should > be compulsory for all research. > > Best wishes, > Peter > > On 6 Jun 2018, at 09:57, Nilesh Madhu <000000405df1884c-dmarc-reques > t@xxxxxxxx> wrote: > > Good morning Bas, > > I see your point and I do agree that, in cases such as those you mention, > pre-defining and getting feedback on the trial procedure you wish to > conduct makes sense. However, for such scenarios, shouldn't there be > industry-wide standards on testing/trials and reporting in place already? > > If someone intentionally conducts a poorly conceived trial, the paper > would/should be rejected in the peer-review anyway ("reject/do not > encourage resubmission"). As I see it, registered reports can be helpful = if > researchers are not sure what trials they should conduct and want early > feedback. Even here, perhaps, a simpler solution would be to make > guidelines available, instead of enforcing an 'administrative' layer. > > Of course, all of this is assuming that the intent is to make registered > reports compulsory for any/every article. If this is not the case, the > point is moot :) > > Greetings from lovely, sunny Belgium (yes, we do have such days!) > > Nilesh > > PS: Massimo, I like your point about the carpenter being adaptive on the > job. Previously I attributed this necessity to adapt to my poor skills ;) > > > > On 2018-06-06 09:35, Bas Van Dijk wrote: > > Hi Nilesh, > I agree to certain extend but I do feel that registered reporting > makes sense for 'close to product' trials and trials that lead to > treatments (for example evaluation of a fitting algorithm). In fact, > it should not really be ' double work' as you fear because if you > execute a poor trial and then try to get it published (believe me.. it > happens :-) )but it gets rejected and you basically have no option but > to redo (part of) the work. (and -re-writing the text to get a poor > trial accepted for publication is of course exactly what you don't > want...). That's more double work that writing up a good trial > proposal, have it reviewed and then know that if you execute according > to plan it's likely to get published even if the results are negative > or non-conclusive, that could be a pre as well. > Best wishes, > Bas > Bas Van Dijk > Program Manager, A&A - Clinician and Research Tools > Cochlear Technology Centre Belgium > <https://maps.google.com/?q=3DBelgium+%0D%0ASchali%C3%ABnhoevedreef+20&en= try=3Dgmail&source=3Dg> > Schali=C3=ABnhoevedreef 20 I > 2800 Mechelen > BELGIUM > Phone: +3215795528 > Mobile: +32473976270 > Email: BVanDijk@xxxxxxxx <BVanDijk@xxxxxxxx> > www.cochlear.com > -----Original Message----- > From: AUDITORY - Research in Auditory Perception > [mailto:AUDITORY@xxxxxxxx <AUDITORY@xxxxxxxx>] On Behalf Of > Nilesh Madhu > Sent: dinsdag 5 juni 2018 13:16 > To: AUDITORY@xxxxxxxx > Subject: [AUDITORY] Registered reports > Dear Tim, > I appreciate your initiative towards reproducible research. However I > fear that registered reports would just add another layer of overhead > to academics and students already under the pressure to publish. If I > understand correctly, this involves two rounds of review: a first > review based on the methodology and evaluation and a second based on > the results of the research. For each stage, probably at least two > review rounds would be needed (going by the current publishing cycle). > I fear, as Gaston does, this might stifle creativity and lead to > overwork also for reviewers and editors. Of course, this is assuming > you want to make registered reports compulsory... > Furthermore, such an approach may not be equally applicable to all > research. For research into algorithms, for example, the value of the > research lies, usually, in the core idea. There are myriad accepted > forms of evaluation and to force a strict evaluation > pattern/methodology would be counterproductive. Reproducible research > in this case is targeted by encouraging authors to make their code and > test data public. > What I would support are (voluntary) guidelines on reporting results > of experiments. This is often to be found in in the engineering field, > when one participates in an open challenge. > Lastly, the main reason for this initiative is to avoid 'mis-reporting' > the results in favour of a hypothesis. Surely, this calls for self > policing? Aren't we, as researchers, possessed of sufficient integrity > and ethics to present our research in the correct light? If this core > value is missing, I fear no external policing is going to help. > Best regards > Nilesh Madhu > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > "The information contained in this e-mail message may be confidential > information, and may also be privileged. If you are not the intended > recipient, any use, interference with, disclosure or copying of this > material is unauthorised and prohibited. If you have received this > message in error, please notify us by return email and delete the > original message." > > > > > -- > *Leslie R. Bernstein, Ph.D. **| *Professor > Depts. of Neuroscience and Surgery (Otolaryngology)| UConn School of > Medicine > 263 Farmington Avenue, Farmington, CT 06030-3401 > <https://maps.google.com/?q=3D263+Farmington+Avenue,+Farmington,+CT+06030= -3401+%0D%0A+Office&entry=3Dgmail&source=3Dg> > Office > <https://maps.google.com/?q=3D263+Farmington+Avenue,+Farmington,+CT+06030= -3401+%0D%0A+Office&entry=3Dgmail&source=3Dg>: > 860.679.4622 | Fax: 860.679.2495 > > > --=20 Julia Strand, PhD Assistant Professor of Psychology Carleton College One North College Street Northfield, MN 55057 507-222-5637 Website <https://apps.carleton.edu/curricular/psyc/jstrand/> Make an appointment <http://juliastrand.youcanbook.me> --0000000000000f889e056e2645c2 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr">Hi all,<div><br></div><div>I'll add one thing to the d= iscussion that hasn't been covered. I recently had my first Stage 1 reg= istered report accepted (in principle) and we've just begun data collec= tion for it. When we went through the first round of peer review, the revie= wers raised several very interesting and important points we hadn't con= sidered. We changed components of the experiment design to reflect their in= put and more cleanly test the question we set out to test, and the experime= nt is much stronger as a result. Had we gone through the process the tradit= ional way, it is entirely possible that the original design would have been= seen as flawed enough not to be publishable (or would require additional e= xperimentation). I know some people are concerned that the RR process would= result in extra work, but this is a case where getting feedback earlier in= the process certainly reduced the amount of work and made the work better = than it otherwise would have been.</div><div><br></div><div>Best,</div><div= >Julia</div></div><div class=3D"gmail_extra"><br><div class=3D"gmail_quote"= >On Fri, Jun 8, 2018 at 2:41 AM, Schoof, Tim <span dir=3D"ltr"><<a href= =3D"mailto:t.schoof@xxxxxxxx" target=3D"_blank">t.schoof@xxxxxxxx</a>>= </span> wrote:<br><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .= 8ex;border-left:1px #ccc solid;padding-left:1ex"> <div dir=3D"ltr"> <div id=3D"m_2373380512428492678divtagdefaultwrapper" style=3D"font-size:12= pt;color:#000000;font-family:Calibri,Helvetica,sans-serif" dir=3D"ltr"> <p style=3D"margin-top:0;margin-bottom:0"></p> <div> <p style=3D"margin-top:0;margin-bottom:0">Dear List,</p> <p style=3D"margin-top:0;margin-bottom:0"><br> </p> <p style=3D"margin-top:0;margin-bottom:0">It has been really great seeing s= uch a lively discussion on the topic!</p> <p style=3D"margin-top:0;margin-bottom:0"><br> </p> <p style=3D"margin-top:0;margin-bottom:0">While it is clear that not everyo= ne sees the benefit of registered reports - and they're probably not pe= rfect, but hopefully they're a step in the right direction - I also kno= w that a lot of people do value them and would be interested in submitting their work as a registered report. I thi= nk it would be good to at least have the option. My impression from psychol= ogy journals that already offer registered reports is that they are not com= pulsory.<br> </p> <p style=3D"margin-top:0;margin-bottom:0"><br> </p> <div>Just over 30 people have contacted me these past few days to say they = would like to sign the <a href=3D"https://osf.io/3wct2/wiki/Journal%20Requests/" rel=3D"noopener n= oreferrer" class=3D"m_2373380512428492678x_OWAAutoLink" id=3D"m_23733805124= 28492678LPlnk901916" target=3D"_blank"> letter</a>. I plan to send this out early next week. If anyone still wants = to add their name to the list, just let me know.</div> <div><br> </div> <div>As a result of this discussion on the auditory list, I know that a few= editors / journals have started thinking and talking about the possibility= of offering registered reports. I'd be more than happy to keep the lis= t updated on any decisions that are made by the various journals in the coming months.<br> </div> <p style=3D"margin-top:0;margin-bottom:0"><br> </p> <p style=3D"margin-top:0;margin-bottom:0">Best,</p> Tim</div> <br> <p></p> <br> <br> <div style=3D"color:rgb(0,0,0)"> <hr style=3D"display:inline-block;width:98%"> <div id=3D"m_2373380512428492678divRplyFwdMsg" dir=3D"ltr"><font style=3D"f= ont-size:11pt" face=3D"Calibri, sans-serif" color=3D"#000000"><b>From:</b> = AUDITORY - Research in Auditory Perception <<a href=3D"mailto:AUDITORY@xxxxxxxx= ISTS.MCGILL.CA" target=3D"_blank">AUDITORY@xxxxxxxx</a>> on behal= f of Les Bernstein <<a href=3D"mailto:lbernstein@xxxxxxxx" target=3D"_bl= ank">lbernstein@xxxxxxxx</a>><br> <b>Sent:</b> 07 June 2018 08:40<br> <b>To:</b> <a href=3D"mailto:AUDITORY@xxxxxxxx" target=3D"_blank">AU= DITORY@xxxxxxxx</a><br> <b>Subject:</b> {SPAM?} Re: Registered reports</font> <div>=C2=A0</div> </div> <div style=3D"background-color:#ffffff"><span class=3D""><font size=3D"-1">= <font face=3D"Verdana">Peter--<br> <br> I think you have benefited the discussion by focusing on the "confirma= tory" vs. "exploratory" CONTINUUM.=C2=A0 Thank you!=C2=A0 I = agree with these two (among others) of your statements:<br> </font></font><br> <font size=3D"-1"><font face=3D"Verdana">1) I think that the debate about r= egistered reports deteriorates too easily into an all-or-nothing argument.<= br> </font></font></span><font size=3D"-1"><font face=3D"Verdana"><span class= =3D"">2) What we need is honesty about where our research falls on this con= tinuum.<br> <br> Despite your recognition of a continuum, your reply references a dichotomou= s view, given your reference to Tukey.=C2=A0 Consider this definition of &q= uot;confirmatory research" from <a class=3D"m_2373380512428492678x_moz-txt-link-freetext m_2373380512428492= 678OWAAutoLink" href=3D"http://www.butlerscientifics.com/single-post/2014/1= 0/08/Exploratory-vs-Confirmatory-Research" id=3D"m_2373380512428492678LPlnk= 601084" target=3D"_blank"> http://www.butlerscientifics.<wbr>com/single-post/2014/10/08/<wbr>Explorato= ry-vs-Confirmatory-<wbr>Research</a>: </span><div id=3D"m_2373380512428492678LPBorder_GT_15284434574320.651891538= 8476153" style=3D"margin-bottom:20px;overflow:auto;width:100%;text-indent:0= px"> <table id=3D"m_2373380512428492678LPContainer_15284434574160.40898598361210= 47" style=3D"width:90%;background-color:rgb(255,255,255);overflow:auto;padd= ing-top:20px;padding-bottom:20px;margin-top:20px;border-top:1px dotted rgb(= 200,200,200);border-bottom:1px dotted rgb(200,200,200)" cellspacing=3D"0"> <tbody> <tr style=3D"border-spacing:0px" valign=3D"top"> <td id=3D"m_2373380512428492678ImageCell_15284434574210.18652971682735642" = style=3D"width:250px;display:table-cell;padding-right:20px" colspan=3D"1"> <div id=3D"m_2373380512428492678LPImageContainer_15284434574210.94253100988= 84251" style=3D"background-color:rgb(255,255,255);height:192px;margin:auto;= display:table;width:192px"> <a id=3D"m_2373380512428492678LPImageAnchor_15284434574240.4642471832518565= 4" style=3D"display:table-cell;text-align:center" href=3D"http://www.butler= scientifics.com/single-post/2014/10/08/Exploratory-vs-Confirmatory-Research= " target=3D"_blank"><img style=3D"display:inline-block;max-width:250px;max-= height:250px;height:192px;width:192px;border-width:0px;vertical-align:botto= m" id=3D"m_2373380512428492678LPThumbnailImageID_15284434574240.69355875943= 81046" width=3D"192" height=3D"192" src=3D"http://static.wixstatic.com/medi= a/625cd8_6c7038b4085c4433623af11549ec6aad.png"></a></div> </td> <td id=3D"m_2373380512428492678TextCell_15284434574280.8516227008042454" st= yle=3D"vertical-align:top;padding:0px;display:table-cell" colspan=3D"2"> <div id=3D"m_2373380512428492678LPRemovePreviewContainer_15284434574280.845= 6838948382911"></div> <div id=3D"m_2373380512428492678LPTitle_15284434574280.9960968851374994" st= yle=3D"color:rgb(0,120,215);font-weight:400;font-size:21px;font-family:&quo= t;wf_segoe-ui_light","Segoe UI Light","Segoe WP Light&q= uot;,"Segoe UI","Segoe WP",Tahoma,Arial,sans-serif;line= -height:21px"> <a id=3D"m_2373380512428492678LPUrlAnchor_15284434574290.4638314355321973" = style=3D"text-decoration:none" href=3D"http://www.butlerscientifics.com/sin= gle-post/2014/10/08/Exploratory-vs-Confirmatory-Research" target=3D"_blank"= >AutoDiscovery Automatic Intelligent Exploratory Data ...</a></div> <div id=3D"m_2373380512428492678LPMetadata_15284434574300.2898083831144924"= style=3D"margin:10px 0px 16px;color:rgb(102,102,102);font-weight:400;font-= family:"wf_segoe-ui_normal","Segoe UI","Segoe WP&q= uot;,Tahoma,Arial,sans-serif;font-size:14px;line-height:14px"> <a href=3D"http://www.butlerscientifics.com" target=3D"_blank">www.butlersc= ientifics.com</a></div> <div id=3D"m_2373380512428492678LPDescription_15284434574310.78316649534302= 87" style=3D"display:block;color:rgb(102,102,102);font-weight:400;font-fami= ly:"wf_segoe-ui_normal","Segoe UI","Segoe WP"= ,Tahoma,Arial,sans-serif;font-size:14px;line-height:20px;max-height:100px;o= verflow:hidden"> AutoDiscovery is an automatic intelligent exploratory data analysis softwar= e that unveils complex relationships in the data files of your experiments.= </div> </td> </tr> </tbody> </table> </div> <br> <br> </font></font><div><div class=3D"h5"><font size=3D"-1"><font face=3D"Verdan= a"><span style=3D"margin:0px;padding:0px;border:0px;outline:0px;vertical-al= ign:baseline;background:0px 0px;color:rgb(89,89,89);font-family:Basic,sans-= serif;font-size:18px;font-style:normal;letter-spacing:normal;text-align:lef= t;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;f= ont-weight:bold">Confirmatory research</span><span style=3D"color:rgb(89,89,89);font-family:Basic,sans-s= erif;font-size:18px;font-style:normal;font-weight:400;letter-spacing:normal= ;text-align:left;text-indent:0px;text-transform:none;white-space:normal;wor= d-spacing:0px;display:inline!important;float:none"><span>=C2=A0</span>(a.k.= a.<span>=C2=A0</span></span><span style=3D"margin:0px;padding:0px;border:0p= x;outline:0px;vertical-align:baseline;background:0px 0px;color:rgb(89,89,89= );font-family:Basic,sans-serif;font-size:18px;font-style:normal;font-weight= :400;letter-spacing:normal;text-align:left;text-indent:0px;text-transform:n= one;white-space:normal;word-spacing:0px;text-decoration:underline"><a href= =3D"https://explorable.com/statistical-hypothesis-testing" rel=3D"undefined= " style=3D"margin:0px;padding:0px;border:0px;outline:0px;vertical-align:bas= eline;background:0px 0px;text-decoration:none;color:inherit" id=3D"m_237338= 0512428492678LPlnk492768" class=3D"m_2373380512428492678OWAAutoLink" target= =3D"_blank">hypothesis testing</a></span><span style=3D"color:rgb(89,89,89);font-family:Basic,san= s-serif;font-size:18px;font-style:normal;font-weight:400;letter-spacing:nor= mal;text-align:left;text-indent:0px;text-transform:none;white-space:normal;= word-spacing:0px;display:inline!important;float:none">) is where researchers have a pretty good idea of what's going on. That = is, researcher has a theory (or several theories), and the objective is to = find out if the theory is supported by the facts.<br> <br> </span></font></font><font size=3D"-1"><font face=3D"Verdana"><span style= =3D"font-family:Basic,sans-serif;font-size:18px;font-style:normal;font-weig= ht:400;letter-spacing:normal;text-align:left;text-indent:0px;text-transform= :none;white-space:normal;word-spacing:0px;display:inline!important;float:no= ne"><font size=3D"-1"><font face=3D"Verdana">While the notion of a continuum is helpful, I would argue that only a very narro= w set of studies are ever merely "confirmatory."=C2=A0 The proble= m lies with the identification of "hypothesis testing" with "= ;confirmatory."=C2=A0 As I see it, it is a false equivalence and is naive.=C2=A0 In many cases a study that seeks to test a hypothesis or hypo= theses falls on your proposed continuum such that a registered reports (RR)= would be undesirable for many of the reasons I identified in previous resp= onses.=C2=A0 As I see it, the set of studies that might be appropriate for RRs are those that offer a virtually unequiv= ocal, binary set of potential outcomes.=C2=A0 Those, in my view, are few an= d far between and are, more often than not, relatively uninteresting.=C2=A0= So, if people wish to use RRs for such studies, then fine.=C2=A0 For the remainder-- most of scientific output-- RRs hold = little value and could, in my view, serve to stifle progress.<br> <br> Les<br> </font></font></span></font></font> <div class=3D"m_2373380512428492678x_moz-cite-prefix"><br> <br> On 6/6/2018 8:04 AM, Peter Harrison wrote:<br> </div> <blockquote type=3D"cite"> <div>Dear list,</div> <div><br> </div> <div>I=E2=80=99ve found this debate very interesting, thank you. Here are s= ome thoughts of my own:</div> <div><br> </div> <div>I think that the debate about registered reports deteriorates too easi= ly into an all-or-nothing argument. Registered reports are ideal for confir= matory research, where it is realistic to specify the analysis in advance, = where the hypothetico-deductive method makes sense, and where the researcher has a good plan of what the f= inal paper should look like before they conduct the study. However, they ar= e often not well-suited to exploratory research where the goal is simply to= find out more about a given phenomenon. As noted already on this thread, in such cases the branching factor of pot= ential analyses may be simply too high to be worth specifying in advance. T= his is particularly true when the researcher wishes to conduct follow-up ex= periments based on the results of previous experiments.</div> <div><br> </div> Confirmatory and exploratory research are both vital - neither one is suffi= cient by=C2=A0itself (see e.g. Turkey 1980=C2=A0-=C2=A0<a href=3D"https://u= rldefense.proofpoint.com/v2/url?u=3Dhttp-3A__www.jstor.org_stable_2682991&a= mp;d=3DDwMGaQ&c=3DEZxp_D7cDnouwj5YEFHgXuSKoUq2zVQZ_7Fw9yfotck&r=3D2= Pw2GwelGcMR4953G-STHGpPJm2-pYYYSPmTwJk3sWM&m=3DQgj_3ZpUHv4jHVkW8H2ur5on= MjbdO0aRXlJg2eqSzb4&s=3DtIRQcUn_AXQVySU6FSm3c0ihAcsI-voUUoP9ji6-mWE&= ;e=3D" class=3D"m_2373380512428492678OWAAutoLink" id=3D"m_23733805124284926= 78LPlnk816638" target=3D"_blank">http://www.jstor.org/<wbr>stable/2682991</= a>). What we need is honesty about where our research falls on this continuum. = Unfortunately the emphasis on the hypothetico-deductive method and null hyp= othesis significance testing in psychological teaching provides many with t= he impression that confirmatory studies with error control are the only =E2=80=98scientific=E2=80=99 way t= o generate knowledge. Perhaps if Bayesian techniques were more commonly tau= ght then we=E2=80=99d be more likely to see the gradual accumulation of pos= itive evidence as a valid alternative. <div><br> </div> <div>On registered reports, then, I believe the following:</div> <div><br> </div> <div>- they should be an good tool for preventing exploratory research from= being published as confirmatory research=C2=A0</div> <div>-> we should encourage journals to offer registered reports as an o= ption</div> <div>-> we should support Tim Schoof in the initiative to write to heari= ng journals</div> <div>-> if you conduct a confirmatory study, then submitting it as a reg= istered report should be a good way to enhance the credibility of your find= ings</div> <div><br> </div> <div>- registered reports are often not suitable for exploratory research= =C2=A0</div> <div>-> we shouldn=E2=80=99t let the impression persist that registered = reports should be compulsory for all research.</div> <div><br> </div> <div>Best wishes,</div> <div>Peter<br> <br> <blockquote type=3D"cite">On 6 Jun 2018, at 09:57, Nilesh Madhu <0000004= 05df1884c-dmarc-<a href=3D"mailto:request@xxxxxxxx" class=3D"m_23733= 80512428492678OWAAutoLink" id=3D"m_2373380512428492678LPlnk332991" target= =3D"_blank">reques<wbr>t@xxxxxxxx</a>> wrote:<br> <br> Good morning Bas,<br> <br> I see your point and I do agree that, in cases such as those you mention, p= re-defining and getting feedback on the trial procedure you wish to conduct= makes=C2=A0sense. However, for such scenarios, shouldn't there be indu= stry-wide standards on=C2=A0testing/trials and reporting in place already?<br> <br> If someone intentionally conducts a poorly conceived trial, the paper would= /should=C2=A0be rejected in the peer-review anyway ("reject/do not enc= ourage resubmission").=C2=A0As I see it, registered reports can be hel= pful if researchers are not sure what trials=C2=A0they should conduct and want early feedback. Even here, perhaps, a simpler=C2=A0soluti= on would be to make guidelines available, instead of enforcing an=C2=A0'= ;administrative' layer.<br> <br> Of course, all of this is assuming that the intent is to make registered re= ports=C2=A0compulsory for any/every article. If this is not the case, the p= oint is moot :)<br> <br> Greetings from lovely, sunny Belgium (yes, we do have such days!)<br> <br> Nilesh<br> <br> PS: Massimo, I like your point about the carpenter being adaptive on the jo= b.=C2=A0Previously I attributed this necessity to adapt to my poor skills ;= )<br> <br> <br> <br> On 2018-06-06 09:35, Bas Van Dijk wrote:<br> <blockquote type=3D"cite">Hi Nilesh,<br> I agree to certain extend but I do feel that registered reporting<br> makes sense for 'close to product' trials and trials that lead to<b= r> treatments (for example evaluation of a fitting algorithm). In fact,<br> it should not really be =C2=A0' double work' as you fear because if= you<br> execute a poor trial and then try to get it published (believe me.. it<br> happens :-) )but it gets rejected and you basically have no option but<br> to redo (part of) the work. (and -re-writing the text to get a poor<br> trial accepted for publication is of course exactly what you don't<br> want...).=C2=A0 That's more double work that writing up a good trial<br= > proposal, have it reviewed and then know that if you execute according<br> to plan it's likely to get published even if the results are negative<b= r> or non-conclusive, that could be a pre as well.<br> Best wishes,<br> Bas<br> Bas Van Dijk<br> Program Manager, A&A - Clinician and Research Tools<br> Cochlear Technology Centre <a href=3D"https://maps.google.com/?q=3DBelgium+= %0D%0ASchali%C3%ABnhoevedreef+20&entry=3Dgmail&source=3Dg">Belgium<= /a><br> Schali=C3=ABnhoevedreef 20 I<br> 2800 Mechelen<br> BELGIUM<br> Phone: +3215795528<br> Mobile: +32473976270<br> <a href=3D"mailto:BVanDijk@xxxxxxxx" class=3D"m_2373380512428492678OWAA= utoLink" id=3D"m_2373380512428492678LPlnk682197" target=3D"_blank">Email: B= VanDijk@xxxxxxxx</a><br> <a class=3D"m_2373380512428492678x_moz-txt-link-abbreviated m_2373380512428= 492678OWAAutoLink" href=3D"http://www.cochlear.com" id=3D"m_237338051242849= 2678LPlnk67861" target=3D"_blank">www.cochlear.com</a><br> -----Original Message-----<br> From: AUDITORY - Research in Auditory Perception<br> [<a class=3D"m_2373380512428492678x_moz-txt-link-freetext m_237338051242849= 2678OWAAutoLink" href=3D"mailto:AUDITORY@xxxxxxxx" id=3D"m_237338051= 2428492678LPlnk925949" target=3D"_blank">mailto:AUDITORY@xxxxxxxx<wbr>= CA</a>] On Behalf Of Nilesh Madhu<br> Sent: dinsdag 5 juni 2018 13:16<br> To: <a class=3D"m_2373380512428492678x_moz-txt-link-abbreviated m_237338051= 2428492678OWAAutoLink" href=3D"mailto:AUDITORY@xxxxxxxx" id=3D"m_237= 3380512428492678LPlnk710703" target=3D"_blank"> AUDITORY@xxxxxxxx</a><br> Subject: [AUDITORY] Registered reports<br> Dear Tim,<br> I appreciate your initiative towards reproducible research. However I<br> fear that registered reports would just add another layer of overhead<br> to academics and students already under the pressure to publish. If I<br> understand correctly, this involves two rounds of review: a first<br> review based on the methodology and evaluation and a second based on<br> the results of the research. For each stage, probably at least two<br> review rounds would be needed (going by the current publishing cycle).<br> I fear, as Gaston does, this might stifle creativity and lead to<br> overwork also for reviewers and editors. Of course, this is assuming<br> you want to make registered reports compulsory...<br> Furthermore, such an approach may not be equally applicable to all<br> research. For research into algorithms, for example, the value of the<br> research lies, usually, in the core idea. There are myriad accepted<br> forms of evaluation and to force a strict evaluation<br> pattern/methodology would be counterproductive. Reproducible research<br> in this case is targeted by encouraging authors to make their code and<br> test data public.<br> What I would support are (voluntary) guidelines on reporting results<br> of experiments. This is often to be found in in the engineering field,<br> when one participates in an open challenge.<br> Lastly, the main reason for this initiative is to avoid 'mis-reporting&= #39;<br> the results in favour of a hypothesis. Surely, this calls for self<br> policing? Aren't we, as researchers, possessed of sufficient integrity<= br> and ethics to present our research in the correct light? If this core<br> value is missing, I fear no external policing is going to help.<br> Best regards<br> Nilesh Madhu<br> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D<wbr>=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D<br> "The information contained in this e-mail message may be confidential<= br> information, and may also be privileged. If you are not the intended<br> recipient, any use, interference with, disclosure or copying of this<br> material is unauthorised and prohibited. If you have received this<br> message in error, please notify us by return email and delete the<br> original message."<br> </blockquote> </blockquote> <br> </div> </blockquote> <br> <br> <div class=3D"m_2373380512428492678x_moz-signature">-- <br> <b><span>Leslie R. Bernstein, Ph.D. </span></b><b><span>| </span> </b><span>Professor</span><span></span><span><br> Depts. of Neuroscience and Surgery (Otolaryngology)| UConn School of Medici= ne </span> <br> <span></span><span><a href=3D"https://maps.google.com/?q=3D263+Farmington+A= venue,+Farmington,+CT+06030-3401+%0D%0A+Office&entry=3Dgmail&source= =3Dg">263 Farmington Avenue, Farmington, CT 06030-3401</a></span><br> <span></span><span><a href=3D"https://maps.google.com/?q=3D263+Farmington+A= venue,+Farmington,+CT+06030-3401+%0D%0A+Office&entry=3Dgmail&source= =3Dg">Office</a>: 860.679.4622 | Fax: 860.679.2495<br> <br> <img alt=3D"" width=3D"125" height=3D"48" src=3D"cid:part5.17ABD5DC.AA24395= 8@xxxxxxxx"><br> </span></div> </div></div></div> </div> </div> </div> </blockquote></div><br><br clear=3D"all"><div><br></div>-- <br><div class= =3D"gmail_signature" data-smartmail=3D"gmail_signature"><div dir=3D"ltr"><d= iv><div dir=3D"ltr">Julia Strand, PhD<div>Assistant Professor of Psychology= </div><div>Carleton College</div><div>One North College Street</div><div>No= rthfield, MN 55057</div><div>507-222-5637</div><div><a href=3D"https://apps= .carleton.edu/curricular/psyc/jstrand/" target=3D"_blank">Website</a></div>= <div><a href=3D"http://juliastrand.youcanbook.me" target=3D"_blank">Make an= appointment</a></div></div></div></div></div> </div> --0000000000000f889e056e2645c2-- --0000000000000f889f056e2645c3 Content-Type: image/png; name="uconnhealth_stacked_blue_email.png" Content-Disposition: inline; filename="uconnhealth_stacked_blue_email.png" Content-Transfer-Encoding: base64 Content-ID: <part5.17ABD5DC.AA243958@xxxxxxxx> X-Attachment-Id: 7472bc98e414a9ba_0.1 iVBORw0KGgoAAAANSUhEUgAAAH0AAAAwCAMAAAALmIWlAAAAGXRFWHRTb2Z0d2FyZQBBZG9iZSBJ bWFnZVJlYWR5ccllPAAAADNQTFRFKzVYHCZM4eLn8PHzpKi30dTbaG+IlZqrs7fDWWB8O0Nkd32U wsXPSlJwhoufDRhA////A68jmAAAABF0Uk5T/////////////////////wAlrZliAAACYklEQVR4 2uzY22KDIAwAUC7e6gjw/187gUC4OetaupflaTXKWSuBKNMuJptC+QMCP20auPHBQCubBWU46K24 uB7NfZ58ZqaMP8D8AEBHtT/goXnB8WPwBf8r1WYUXWxUOZobHcJXIMcfONclmE4s0lrRzYBIOjvT jX5WV6YbP2ZQJ6TWjfiAbuYzHW7p+xpuolQrP83sjQ5nupnu6LqaJ92MbnSznulcfkBHpNXNY6y+ c0JKnWX1eFeflQt5qcNESKnrQMrf6LSAXeiWkEoXdOU4nZBKxzPnoTohtW5ZZyK8W09Io6tYjyP1 iDS6XbAeh815S0irS6zHUfXuB0Ok1S3W41AdEWh1/Nb7UD1bWytd0LY4ThdnerYbBJ1taT5XBmy4 J8mNnWagp5fbba6HeiTdneK6vNgoFj0EhyO4eSZT6AlpdNXoNzqYi94GbIU0eqhHp3d5drRf4heZ Uo9Iq8vYptup6VB37H6m/W6m0hFpdaxH3/yK9cHpVq5Zy59n2GMV/QxPGdV9NElPE9njyOozsfWW Kq6j7bLuw97KPBnM/mX86//6n+tH/yLKP7AUXcyd80Lx53WnspB47ty5qKND1gLockfIFzHR7srp Q7X0+6GgHPNFXedPyaN1CPFli71z+ZBeTxbcI2RXxwOQj/lW/Ui610Z6gA4a32WRrssXYbO76V/Z y6FLnZVj/qCnqGcdZE1DeFKcntWrMV/RZXCh7B/fpF/+8tQQx6J74y9/Oeuyrm75+Jyf8oKWH9CL 1SYNJtJ91HTOAD2fdjPN9UcsumwmDNYXqvPtOLa9pH8LMACnoV0siZAyOAAAAABJRU5ErkJggg== --0000000000000f889f056e2645c3--