[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [AUDITORY] Registered reports

Les,


Thanks for your response.  Note that I mentioned a number of issues that
I identify as problems and shortcomings and not just a single one.  That
the "results" section of a Stage 2 submission allows for "exploratory
analysis" hardly addresses the issues I raised with regard to
hypotheses, follow-up and control experiments, choice of PRIMARY
statistical tests, and archival value.  Furthermore, the "exploratory"
analyses you cite are clearly considered subordinate to the pre-approved
"confirmatory analyses."  See Nosek and Lakens (2014).  As I see it,
that's unnecessarily restrictive.


- Multiple experiments. Usually, Experiment 2 follows the results of
Experiment 1 (that in RR are unknown). One solution is that you do a
registered report of Experiment 1 and a new registered report for
Experiment 2
- confirmatory vs exploratory analysis. This is exactly the point.
Nowadays often we sell "exploratory" like "confirmatory". In contrast,
we should make clear what is exploratory and what is not. I think nobody
would ignore an interesting exploratory result.
- archival value. Nobody knows whether the "success rate" of RR is
higher equal or lower than traditional paper. Perhaps there are not many
data yet. But we do know that current literature is inflated with false
positive (e.g., look at the various replication experiments or at the
analysis by Ioannidis et al.). In my opinion, the auditory field is a
safe island (at least in comparison to other fields). However, I have no
data about it.


Yes, we do "move more like a carpenter trying to adapt and adjust things
in real time." Registered reports ask that we plan most, if not all, of
our measurements, cuts, and adjustments in advance. They are anathema to
the process.


I would like to move from "carpenter" to "engineer" :-)
RR and other standards that are now suggested (e.g., preregistration,
Bayesian stats, multi-lab experiments) enable to do so (in my opinion).
In any case, all journals that are adopting RR still offer the
traditional submissions. So everything is preserved.


I agree with Nilesh's comments, especially, "Aren't we, as researchers,
possessed of sufficient integrity and ethics to present our research in
the correct light? If this core value is missing, I fear no external
policing is going to help."


- I don't know. For example, here in Italy -where staff recruiting is
screened by number of publications, H-index and number of citations-
researchers are pushed a lot in the direction of "publish as much as
possible in high impact factor journals and get cited a lot or perish".
And in fact a recent editorial in Nature was highlighting that here in
Italy the number of self-citations is increasing. I'm wandering whether
others not-so-nice behaviours are also adopted (Interesting enough,
Italy still scores zero (!) for number of scientific frauds. There was
list in wikipedia, I can't find it, sorry.)

If we look back at the history of psychology (my own field), it looks to
me we are facing a change in scientific paradigm. From Wundt up to
Titchener we were using (and trusting) introspection as a good tool to
investigate psychology. Then we had the paper by John Watson (1913,
Psychology as the behaviorist views it. Psychological review, 20(2),
158-177.) and in a few years time introspection was forgotten and
neglected. Let's look back at Watson's statement now: it sounds so
obvious today ("Introspection forms no essential part of its methods,
nor is the scientific value of its data dependent upon the readiness
with which they lend themselves to interpretation in terms of
consciousness.). My guess is that in a few years time we will do the
same for several of the current research practices.

Apologies for the long email and ll the best from a scorchy hot Italy,

m

ps: no carpenter has been killed or injured while writing this email.