[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [AUDITORY] [Non-DoD Source] Re: Registered reports (UNCLASSIFIED)
CLASSIFICATION: UNCLASSIFIED
THIS: "- about the possible "uncontrolled dissemination of null results", I think that (for science) the current uncontrolled dissemination of type I errors is worse."
I think that there is something to be said for a repository of experimental results regardless of publication. I suspect that there are many papers that go unpublished for lack of significant results - or that have questionable statistics applied due to failure to find significance using good statistical practices and the need to publish.
That said, my own organization has a fairly rigorous process for IRB approval that requires several levels of review and demands specific details for planned statistical analysis. It takes me longer to begin a study than to complete and write it up! I would probably not want to undergo a second external procedure.
A.
-----Original Message-----
From: AUDITORY - Research in Auditory Perception [mailto:AUDITORY@xxxxxxxxxxxxxxx] On Behalf Of Massimo Grassi
Sent: Wednesday, June 6, 2018 2:10 AM
To: AUDITORY@xxxxxxxxxxxxxxx
Subject: [Non-DoD Source] Re: Registered reports
All active links contained in this email were disabled. Please verify the identity of the sender, and confirm the authenticity of all links contained within the message prior to copying and pasting the address to a Web browser.
----
First of all I would like to thank Tim for the initiative.
A few replies and comments:
- registered reports have the results section divided in the parts: the "planned analysis" (those you discussed with editor and reviewers) and the "new exploratory analysis". Therefore, I do not see the problem risen by Les.
- in my opinion registered reports rise the standard level of current science. Registered reports (like a preregistration but even better) reveal how limited is our ability to predict. It is difficult to predict how the data will look like, what data point will be an outlier, whether data should be analysed in this or that way. We teach to students that the path of science is hypothetical deductive. In reality we move more like a carpenter trying to adapt and adjust things in real time.
- about the possible "uncontrolled dissemination of null results", I think that (for science) the current uncontrolled dissemination of type I errors is worse.
A nice day to everybody from a summer-sunny Italy,
m
> Dear List,
>
> For this topic, I'll violate my rule of not posting replies here. I
> agree with Ms. Rankovic. I sure did not miss the substance and detail
> of Mr. Schoof's email. I also read over the information in the links.
> Indeed, the proposed plan provides for a second review. It seems to
> me, however, that the provisional acceptance is a key aspect of the
> process. If it were the case that manuscripts were rejected upon
> second review with substantial frequency, then the philosophy of the
> registered report would be violated and the system would collapse.
> So, unless there are egregious errors or flaws in the full manuscript,
> it seems that it would be published. Note that, in this linked
> reference <Caution-https://orca.cf.ac.uk/59475/1/AN2.pdf>, publication
> is assumed to be "guaranteed."
>
> In my opinion, the criticism found within the FAQ here
> <Caution-https://cos.io/rr/>, that "The Registered Reports model is
> based on a naïve conceptualisation of the scientific method." is
> well-founded! The reply offered to counter that criticism is quite weak and unconvincing.
> I would replace "scientific method" in that criticism with "the way
> good science is done."
>
> Question 17 in Chambers et al. (2014-- linked above) provides an apt
> example. In the process of conducting complex experiments, it is very
> often the case that unexpected results lead to important follow-up or
> control experiments. Chambers et al. handle this issue by proposing
> that in Stage 1 of a registered report, contingencies be stated such
> that "If A is observed, then we will..." That, of course, assumes
> that one knows the decision tree in advance! In my experience,
> science simply does not work that way.
>
> While I find the intent of registered reports to be laudable, in my
> opinion, it substitutes one potential set of problems with another
> based on a narrow view of how science proceeds. Indeed, one may have
> a hypothesis to be tested and gather a set of data to address it only
> to find that the results support a substantially altered view. Is
> that, NECESSARILY, the dreaded "HARKing?" I think not. Scientific
> thought and inquiry do not always proceed in a linear fashion. One
> cannot and should not always know the precise questions or list of
> contingencies a priori and be restricted to answering only those.
> Then there are experiments in which there are no specific hypotheses.
> They may be of the form, "What is the effect of variable A on measurements of X?"
> Assuming the question is non-trivial, those are often the most
> revealing experiments because any outcome is of interest. There is no "positive"
> or "negative." Sure, one can cast such experiments in terms of
> hypotheses but doing so often involves a contrivance.
>
> Then there is the matter of "p-hacking" and what I would call
> "statistics shopping." Indeed, it is a problem. Unexpected outcomes
> and patterns of data in a complex experiment often require one to
> choose the appropriate statistic after the fact. It is sometimes the
> correct thing to do! Whether it is proper can and should be judged by
> reviewers with the requisite expertise. Good peer-review should
> distinguish between p-hacking and a rational choice that conveys
> information and "truth." The notion that one can and should use only
> the statistic decided upon in advance is unnecessary restrictive.
>
> Finally, there is the matter of archival value. According to Chambers
> et al., "...if the rationale and methods are sound then the journal
> should agree to publish the final paper regardless of the specific
> outcome." It is often the case that rationale and methods are sound
> but the data provide no substantial advance or archival value. I'm
> not sure that "approving" a method and rationale and virtually
> guaranteeing publication will afford the same level of judgment in
> terms of archival value that is afforded by the current system.
>
> Les Bernstein
>
> --
> *Leslie R. Bernstein, Ph.D. **| *Professor Depts. of Neuroscience and
> Surgery (Otolaryngology)| UConn School of Medicine
> 263 Farmington Avenue, Farmington, CT 06030-3401
> Office: 860.679.4622 | Fax: 860.679.2495
>
>
>
>
>
>
>
> <Caution-https://urldefense.proofpoint.com/v2/url?u=https-3A__cos.io_r
> r_&d=DwMFAg&c=EZxp_D7cDnouwj5YEFHgXuSKoUq2zVQZ_7Fw9yfotck&r=2Pw2GwelGc
> MR4953G-STHGpPJm2-pYYYSPmTwJk3sWM&m=Sr0Ep-Gx1c9KJlrgGBL4rmcUvd9qeDUDnF
> KUymDoKpI&s=vXqZBKaP1dUovPzwBwC5DalLCB6UxwKuM9x_SQCbw5I&e=>
>
> On 6/4/2018 7:51 AM, Christine Rankovic wrote:
>>
>> Mr. Schoof:
>>
>> It is beyond ridiculous to accept partial manuscripts for publication.
>>
>> Christine Rankovic, PhD
>>
>> Scientist, Speech and Hearing
>>
>> Newton, MA USA
>>
>> rankovic@xxxxxxxxxxxxxxxx
>>
>> *From:*AUDITORY - Research in Auditory Perception
>> [Caution-mailto:AUDITORY@xxxxxxxxxxxxxxx] *On Behalf Of *Schoof, Tim
>> *Sent:* Monday, June 04, 2018 4:06 AM
>> *To:* AUDITORY@xxxxxxxxxxxxxxx
>> *Subject:* Registered reports
>>
>> Dear list,
>>
>> I'm going to try and get hearing science journals to start offering
>> registered reports. These reports are basically peer-reviewed
>> pre-registration documents where you outline your methods and
>> proposed analyses. If this document makes it through peer-review, the
>> manuscript is provisionally accepted for publication. This process
>> should reduce certain questionable research practices, such as
>> selective reporting of results and publication bias. If you're
>> sceptical about registered reports, the Center for Open Science has
>> compiled a nice FAQ list that might address some of your concerns:
>> Caution-https://cos.io/rr/
>> <Caution-https://urldefense.proofpoint.com/v2/url?u=https-3A__cos.io_
>> rr_&d=DwMFAg&c=EZxp_D7cDnouwj5YEFHgXuSKoUq2zVQZ_7Fw9yfotck&r=2Pw2Gwel
>> GcMR4953G-STHGpPJm2-pYYYSPmTwJk3sWM&m=Sr0Ep-Gx1c9KJlrgGBL4rmcUvd9qeDU
>> DnFKUymDoKpI&s=vXqZBKaP1dUovPzwBwC5DalLCB6UxwKuM9x_SQCbw5I&e=>
>>
>> I think this is the direction science is going in now and it would be
>> great if hearing science joined in. I plan to contact as many hearing
>> science journals as possible. I'm compiling a list of journals to
>> contact. Please add to this list if I'm missing anything:
>> Caution-https://tinyurl.com/yaf9r7bk
>> <Caution-https://urldefense.proofpoint.com/v2/url?u=https-3A__tinyurl.com_yaf9r7bk&d=DwMFAg&c=EZxp_D7cDnouwj5YEFHgXuSKoUq2zVQZ_7Fw9yfotck&r=2Pw2GwelGcMR4953G-STHGpPJm2-pYYYSPmTwJk3sWM&m=Sr0Ep-Gx1c9KJlrgGBL4rmcUvd9qeDUDnFKUymDoKpI&s=sk2rFf3fImx-wI9S05uLc7WYgADb5BupEMAQvL3hz-0&e=>.
>> I don't think any of these journals offer (or are in the process of
>> offering) registered reports yet, but correct me if I'm wrong.
>>
>> If you agree that registered reports are a good idea and want to sign
>> the letter I intend to send (see here for a template:
>> Caution-https://osf.io/3wct2/wiki/Journal%20Requests/
>> <Caution-https://urldefense.proofpoint.com/v2/url?u=https-3A__osf.io_
>> 3wct2_wiki_Journal-2520Requests_&d=DwMFAg&c=EZxp_D7cDnouwj5YEFHgXuSKo
>> Uq2zVQZ_7Fw9yfotck&r=2Pw2GwelGcMR4953G-STHGpPJm2-pYYYSPmTwJk3sWM&m=Sr
>> 0Ep-Gx1c9KJlrgGBL4rmcUvd9qeDUDnFKUymDoKpI&s=G-jhAt3_0f5cPPX7aRpPgVfih
>> ZYm_ZTuPohnhVfxWFw&e=>), let me know and I'll add you to the list.
>> And please spread the word of course. The more people agree, the more
>> likely it is we can get some of these journals on board!
>>
>> Best,
>>
>>
>> Tim Schoof
>>
>> --
>>
>> Research Associate
>>
>> UCL Speech, Hearing and Phonetic Sciences
>>
>> Chandler House
>>
>> 2 Wakefield Street
>>
>> London WC1N 1PF
>>
>> United Kingdom
>>
>
>
>
CLASSIFICATION: UNCLASSIFIED