Re: [AUDITORY] Registered reports

[Les Bernstein <lbernstein@xxxxxxxx>]

Peter--

I think you have benefited the discussion by focusing on the "confirmatory" vs. "exploratory" CONTINUUM.  Thank you!  I agree with these two (among others) of your statements:

1) I think that the debate about registered reports deteriorates too easily into an all-or-nothing argument.
2) What we need is honesty about where our research falls on this continuum.

Despite your recognition of a continuum, your reply references a dichotomous view, given your reference to Tukey.  Consider this definition of "confirmatory research" from http://www.butlerscientifics.com/single-post/2014/10/08/Exploratory-vs-Confirmatory-Research:

Confirmatory research (a.k.a. hypothesis testing) is where researchers have a pretty good idea of what's going on. That is, researcher has a theory (or several theories), and the objective is to find out if the theory is supported by the facts.

While the notion of a continuum is helpful, I would argue that only a very narrow set of studies are ever merely "confirmatory."  The problem lies with the identification of "hypothesis testing" with "confirmatory."  As I see it, it is a false equivalence and is naive.  In many cases a study that seeks to test a hypothesis or hypotheses falls on your proposed continuum such that a registered reports (RR) would be undesirable for many of the reasons I identified in previous responses.  As I see it, the set of studies that might be appropriate for RRs are those that offer a virtually unequivocal, binary set of potential outcomes.  Those, in my view, are few and far between and are, more often than not, relatively uninteresting.  So, if people wish to use RRs for such studies, then fine.  For the remainder-- most of scientific output-- RRs hold little value and could, in my view, serve to stifle progress.

Les

On 6/6/2018 8:04 AM, Peter Harrison wrote:

Dear list,

I’ve found this debate very interesting, thank you. Here are some thoughts of my own:

I think that the debate about registered reports deteriorates too easily into an all-or-nothing argument. Registered reports are ideal for confirmatory research, where it is realistic to specify the analysis in advance, where the hypothetico-deductive method makes sense, and where the researcher has a good plan of what the final paper should look like before they conduct the study. However, they are often not well-suited to exploratory research where the goal is simply to find out more about a given phenomenon. As noted already on this thread, in such cases the branching factor of potential analyses may be simply too high to be worth specifying in advance. This is particularly true when the researcher wishes to conduct follow-up experiments based on the results of previous experiments.

Confirmatory and exploratory research are both vital - neither one is sufficient by itself (see e.g. Turkey 1980 - http://www.jstor.org/stable/2682991). What we need is honesty about where our research falls on this continuum. Unfortunately the emphasis on the hypothetico-deductive method and null hypothesis significance testing in psychological teaching provides many with the impression that confirmatory studies with error control are the only ‘scientific’ way to generate knowledge. Perhaps if Bayesian techniques were more commonly taught then we’d be more likely to see the gradual accumulation of positive evidence as a valid alternative.

On registered reports, then, I believe the following:

- they should be an good tool for preventing exploratory research from being published as confirmatory research 

-> we should encourage journals to offer registered reports as an option

-> we should support Tim Schoof in the initiative to write to hearing journals

-> if you conduct a confirmatory study, then submitting it as a registered report should be a good way to enhance the credibility of your findings

- registered reports are often not suitable for exploratory research 

-> we shouldn’t let the impression persist that registered reports should be compulsory for all research.

Best wishes,

Peter

On 6 Jun 2018, at 09:57, Nilesh Madhu <000000405df1884c-dmarc-request@xxxxxxxxxxxxxxx> wrote:

Good morning Bas,

I see your point and I do agree that, in cases such as those you mention, pre-defining and getting feedback on the trial procedure you wish to conduct makes sense. However, for such scenarios, shouldn't there be industry-wide standards on testing/trials and reporting in place already?

If someone intentionally conducts a poorly conceived trial, the paper would/should be rejected in the peer-review anyway ("reject/do not encourage resubmission"). As I see it, registered reports can be helpful if researchers are not sure what trials they should conduct and want early feedback. Even here, perhaps, a simpler solution would be to make guidelines available, instead of enforcing an 'administrative' layer.

Of course, all of this is assuming that the intent is to make registered reports compulsory for any/every article. If this is not the case, the point is moot :)

Greetings from lovely, sunny Belgium (yes, we do have such days!)

Nilesh

PS: Massimo, I like your point about the carpenter being adaptive on the job. Previously I attributed this necessity to adapt to my poor skills ;)



On 2018-06-06 09:35, Bas Van Dijk wrote:

Hi Nilesh,
I agree to certain extend but I do feel that registered reporting
makes sense for 'close to product' trials and trials that lead to
treatments (for example evaluation of a fitting algorithm). In fact,
it should not really be  ' double work' as you fear because if you
execute a poor trial and then try to get it published (believe me.. it
happens :-) )but it gets rejected and you basically have no option but
to redo (part of) the work. (and -re-writing the text to get a poor
trial accepted for publication is of course exactly what you don't
want...).  That's more double work that writing up a good trial
proposal, have it reviewed and then know that if you execute according
to plan it's likely to get published even if the results are negative
or non-conclusive, that could be a pre as well.
Best wishes,
Bas
Bas Van Dijk
Program Manager, A&A - Clinician and Research Tools
Cochlear Technology Centre Belgium
Schaliënhoevedreef 20 I
2800 Mechelen
BELGIUM
Phone: +3215795528
Mobile: +32473976270
Email: BVanDijk@xxxxxxxxxxxx
www.cochlear.com
-----Original Message-----
From: AUDITORY - Research in Auditory Perception
[mailto:AUDITORY@xxxxxxxxxxxxxxx] On Behalf Of Nilesh Madhu
Sent: dinsdag 5 juni 2018 13:16
To: AUDITORY@xxxxxxxxxxxxxxx
Subject: [AUDITORY] Registered reports
Dear Tim,
I appreciate your initiative towards reproducible research. However I
fear that registered reports would just add another layer of overhead
to academics and students already under the pressure to publish. If I
understand correctly, this involves two rounds of review: a first
review based on the methodology and evaluation and a second based on
the results of the research. For each stage, probably at least two
review rounds would be needed (going by the current publishing cycle).
I fear, as Gaston does, this might stifle creativity and lead to
overwork also for reviewers and editors. Of course, this is assuming
you want to make registered reports compulsory...
Furthermore, such an approach may not be equally applicable to all
research. For research into algorithms, for example, the value of the
research lies, usually, in the core idea. There are myriad accepted
forms of evaluation and to force a strict evaluation
pattern/methodology would be counterproductive. Reproducible research
in this case is targeted by encouraging authors to make their code and
test data public.
What I would support are (voluntary) guidelines on reporting results
of experiments. This is often to be found in in the engineering field,
when one participates in an open challenge.
Lastly, the main reason for this initiative is to avoid 'mis-reporting'
the results in favour of a hypothesis. Surely, this calls for self
policing? Aren't we, as researchers, possessed of sufficient integrity
and ethics to present our research in the correct light? If this core
value is missing, I fear no external policing is going to help.
Best regards
Nilesh Madhu
==============================================
"The information contained in this e-mail message may be confidential
information, and may also be privileged. If you are not the intended
recipient, any use, interference with, disclosure or copying of this
material is unauthorised and prohibited. If you have received this
message in error, please notify us by return email and delete the
original message."

--
Leslie R. Bernstein, Ph.D. | Professor
Depts. of Neuroscience and Surgery (Otolaryngology)| UConn School of Medicine
263 Farmington Avenue, Farmington, CT 06030-3401
Office: 860.679.4622 | Fax: 860.679.2495