Subject: Re: [AUDITORY] [Non-DoD Source] Re: Registered reports (UNCLASSIFIED) From: US <"Scharine, Angelique A CIV USARMY RDECOM ARL "> Date: Wed, 6 Jun 2018 11:34:49 +0000 List-Archive:<http://lists.mcgill.ca/scripts/wa.exe?LIST=AUDITORY>CLASSIFICATION: UNCLASSIFIED THIS: "- about the possible "uncontrolled dissemination of null results", I= think that (for science) the current uncontrolled dissemination of type I = errors is worse." I think that there is something to be said for a repository of experimental= results regardless of publication. I suspect that there are many papers th= at go unpublished for lack of significant results - or that have questionab= le statistics applied due to failure to find significance using good statis= tical practices and the need to publish.=20 That said, my own organization has a fairly rigorous process for IRB approv= al that requires several levels of review and demands specific details for = planned statistical analysis. It takes me longer to begin a study than to c= omplete and write it up! I would probably not want to undergo a second exte= rnal procedure.=20 A. -----Original Message----- From: AUDITORY - Research in Auditory Perception [mailto:AUDITORY@xxxxxxxx= ILL.CA] On Behalf Of Massimo Grassi Sent: Wednesday, June 6, 2018 2:10 AM To: AUDITORY@xxxxxxxx Subject: [Non-DoD Source] Re: Registered reports All active links contained in this email were disabled. Please verify the = identity of the sender, and confirm the authenticity of all links contained= within the message prior to copying and pasting the address to a Web brows= er. =20 ---- First of all I would like to thank Tim for the initiative. A few replies and comments: - registered reports have the results section divided in the parts: the "pl= anned analysis" (those you discussed with editor and reviewers) and the "ne= w exploratory analysis". Therefore, I do not see the problem risen by Les. - in my opinion registered reports rise the standard level of current scien= ce. Registered reports (like a preregistration but even better) reveal how = limited is our ability to predict. It is difficult to predict how the data = will look like, what data point will be an outlier, whether data should be = analysed in this or that way. We teach to students that the path of science= is hypothetical deductive. In reality we move more like a carpenter trying= to adapt and adjust things in real time. - about the possible "uncontrolled dissemination of null results", I think = that (for science) the current uncontrolled dissemination of type I errors = is worse. A nice day to everybody from a summer-sunny Italy, m > Dear List, > > For this topic, I'll violate my rule of not posting replies here.=A0 I=20 > agree with Ms. Rankovic.=A0 I sure did not miss the substance and detail= =20 > of Mr. Schoof's email.=A0 I also read over the information in the links. > Indeed, the proposed plan provides for a second review.=A0 It seems to=20 > me, however, that the provisional acceptance is a key aspect of the=20 > process.=A0 If it were the case that manuscripts were rejected upon=20 > second review with substantial frequency, then the philosophy of the=20 > registered report would be violated and the system would collapse.=A0=20 > So, unless there are egregious errors or flaws in the full manuscript,=20 > it seems that it would be published.=A0 Note that, in this linked=20 > reference <Caution-https://orca.cf.ac.uk/59475/1/AN2.pdf>, publication=20 > is assumed to be "guaranteed." > > In my opinion, the criticism found within the FAQ here=20 > <Caution-https://cos.io/rr/>, that "The Registered Reports model is=20 > based on a na=EFve conceptualisation of the scientific method." is=20 > well-founded! The reply offered to counter that criticism is quite weak a= nd unconvincing. > I would replace "scientific method" in that criticism with "the way=20 > good science is done." > > Question 17 in Chambers et al. (2014-- linked above) provides an apt=20 > example.=A0 In the process of conducting complex experiments, it is very= =20 > often the case that unexpected results lead to important follow-up or=20 > control experiments.=A0 Chambers et al. handle this issue by proposing=20 > that in Stage 1 of a registered report, contingencies be stated such=20 > that "If A is observed, then we will..."=A0 That, of course, assumes=20 > that one knows the decision tree in advance!=A0 In my experience,=20 > science simply does not work that way. > > While I find the intent of registered reports to be laudable, in my=20 > opinion, it substitutes one potential set of problems with another=20 > based on a narrow view of how science proceeds.=A0 Indeed, one may have=20 > a hypothesis to be tested and gather a set of data to address it only=20 > to find that the results support a substantially altered view.=A0 Is=20 > that, NECESSARILY, the dreaded "HARKing?"=A0 I think not.=A0 Scientific=20 > thought and inquiry do not always proceed in a linear fashion.=A0 One=20 > cannot and should not always know the precise questions or list of=20 > contingencies a priori and be restricted to answering only those.=A0=20 > Then there are experiments in which there are no specific hypotheses.=A0= =20 > They may be of the form, "What is the effect of variable A on measurement= s of X?" > Assuming the question is non-trivial, those are often the most=20 > revealing experiments because any outcome is of interest.=A0 There is no = "positive" > or "negative."=A0 Sure, one can cast such experiments in terms of=20 > hypotheses but doing so often involves a contrivance. > > Then there is the matter of "p-hacking" and what I would call=20 > "statistics shopping."=A0 Indeed, it is a problem.=A0 Unexpected outcomes= =20 > and patterns of data in a complex experiment often require one to=20 > choose the appropriate statistic after the fact. It is sometimes the=20 > correct thing to do!=A0 Whether it is proper can and should be judged by= =20 > reviewers with the requisite expertise.=A0 Good peer-review should=20 > distinguish between p-hacking and a rational choice that conveys=20 > information and "truth."=A0 The notion that one can and should use only=20 > the statistic decided upon in advance is unnecessary restrictive. > > Finally, there is the matter of archival value.=A0 According to Chambers= =20 > et al., "...if the rationale and methods are sound then the journal=20 > should agree to publish the final paper regardless of the specific=20 > outcome."=A0 It is often the case that rationale and methods are sound=20 > but the data provide no substantial advance or archival value.=A0 I'm=20 > not sure that "approving" a method and rationale and virtually=20 > guaranteeing publication will afford the same level of judgment in=20 > terms of archival value that is afforded by the current system. > > Les Bernstein > > -- > *Leslie R. Bernstein, Ph.D. **| *Professor Depts. of Neuroscience and=20 > Surgery (Otolaryngology)| UConn School of Medicine > 263 Farmington Avenue, Farmington, CT 06030-3401 > Office: 860.679.4622 | Fax: 860.679.2495 > > > > > > > > <Caution-https://urldefense.proofpoint.com/v2/url?u=3Dhttps-3A__cos.io_r > r_&d=3DDwMFAg&c=3DEZxp_D7cDnouwj5YEFHgXuSKoUq2zVQZ_7Fw9yfotck&r=3D2Pw2Gwe= lGc > MR4953G-STHGpPJm2-pYYYSPmTwJk3sWM&m=3DSr0Ep-Gx1c9KJlrgGBL4rmcUvd9qeDUDnF > KUymDoKpI&s=3DvXqZBKaP1dUovPzwBwC5DalLCB6UxwKuM9x_SQCbw5I&e=3D> > > On 6/4/2018 7:51 AM, Christine Rankovic wrote: >> >> Mr. Schoof: >> >> It is beyond ridiculous to accept partial manuscripts for publication. >> >> Christine Rankovic, PhD >> >> Scientist, Speech and Hearing >> >> Newton, MA=A0 USA >> >> rankovic@xxxxxxxx >> >> *From:*AUDITORY - Research in Auditory Perception=20 >> [Caution-mailto:AUDITORY@xxxxxxxx *On Behalf Of *Schoof, Tim >> *Sent:* Monday, June 04, 2018 4:06 AM >> *To:* AUDITORY@xxxxxxxx >> *Subject:* Registered reports >> >> Dear list, >> >> I'm going to try and get hearing science journals to start offering=20 >> registered reports. These reports are basically peer-reviewed=20 >> pre-registration documents where you outline your methods and=20 >> proposed analyses. If this document makes it through peer-review, the=20 >> manuscript is provisionally accepted for publication. This process=20 >> should reduce certain questionable research practices, such as=A0 >> selective reporting of results and publication bias. If you're=20 >> sceptical about registered reports,=A0the Center for Open Science has=20 >> compiled a nice FAQ list that might address some of your concerns: >> Caution-https://cos.io/rr/ >> <Caution-https://urldefense.proofpoint.com/v2/url?u=3Dhttps-3A__cos.io_ >> rr_&d=3DDwMFAg&c=3DEZxp_D7cDnouwj5YEFHgXuSKoUq2zVQZ_7Fw9yfotck&r=3D2Pw2G= wel >> GcMR4953G-STHGpPJm2-pYYYSPmTwJk3sWM&m=3DSr0Ep-Gx1c9KJlrgGBL4rmcUvd9qeDU >> DnFKUymDoKpI&s=3DvXqZBKaP1dUovPzwBwC5DalLCB6UxwKuM9x_SQCbw5I&e=3D> >> >> I think this is the direction science is going in now and it would be=20 >> great if hearing science joined in. I plan to contact as many hearing=20 >> science journals as possible. I'm compiling a list of journals to=20 >> contact. Please add to this list if I'm missing anything: >> Caution-https://tinyurl.com/yaf9r7bk >> <Caution-https://urldefense.proofpoint.com/v2/url?u=3Dhttps-3A__tinyurl.= com_yaf9r7bk&d=3DDwMFAg&c=3DEZxp_D7cDnouwj5YEFHgXuSKoUq2zVQZ_7Fw9yfotck&r= =3D2Pw2GwelGcMR4953G-STHGpPJm2-pYYYSPmTwJk3sWM&m=3DSr0Ep-Gx1c9KJlrgGBL4rmcU= vd9qeDUDnFKUymDoKpI&s=3Dsk2rFf3fImx-wI9S05uLc7WYgADb5BupEMAQvL3hz-0&e=3D>. >> I don't think any of these journals offer (or are in the process of >> offering) registered reports yet, but correct me if I'm wrong. >> >> If you agree that registered reports are a good idea and want to sign=20 >> the letter I intend to send (see here for a template: >> Caution-https://osf.io/3wct2/wiki/Journal%20Requests/ >> <Caution-https://urldefense.proofpoint.com/v2/url?u=3Dhttps-3A__osf.io_ >> 3wct2_wiki_Journal-2520Requests_&d=3DDwMFAg&c=3DEZxp_D7cDnouwj5YEFHgXuSK= o >> Uq2zVQZ_7Fw9yfotck&r=3D2Pw2GwelGcMR4953G-STHGpPJm2-pYYYSPmTwJk3sWM&m=3DS= r >> 0Ep-Gx1c9KJlrgGBL4rmcUvd9qeDUDnFKUymDoKpI&s=3DG-jhAt3_0f5cPPX7aRpPgVfih >> ZYm_ZTuPohnhVfxWFw&e=3D>), let me know and I'll add you to the list.=20 >> And please spread the word of course. The more people agree, the more=20 >> likely it is we can get some of these journals on board! >> >> Best, >> >> >> Tim Schoof >> >> -- >> >> Research Associate >> >> UCL Speech, Hearing and Phonetic Sciences >> >> Chandler House >> >> 2 Wakefield Street >> >> London WC1N 1PF >> >> United Kingdom >> > > > CLASSIFICATION: UNCLASSIFIED