Subject: Re: [AUDITORY] Registered reports From: Massimo Grassi <massimo.grassi@xxxxxxxx> Date: Wed, 6 Jun 2018 18:15:26 +0200 List-Archive:<http://lists.mcgill.ca/scripts/wa.exe?LIST=AUDITORY>Les, > Thanks for your response.=A0 Note that I mentioned a number of issues t= hat > I identify as problems and shortcomings and not just a single one.=A0 T= hat > the "results" section of a Stage 2 submission allows for "exploratory > analysis" hardly addresses the issues I raised with regard to > hypotheses, follow-up and control experiments, choice of PRIMARY > statistical tests, and archival value.=A0 Furthermore, the "exploratory= " > analyses you cite are clearly considered subordinate to the pre-approve= d > "confirmatory analyses."=A0 See Nosek and Lakens (2014).=A0 As I see it= , > that's unnecessarily restrictive. - Multiple experiments. Usually, Experiment 2 follows the results of Experiment 1 (that in RR are unknown). One solution is that you do a registered report of Experiment 1 and a new registered report for Experiment 2 - confirmatory vs exploratory analysis. This is exactly the point. Nowadays often we sell "exploratory" like "confirmatory". In contrast, we should make clear what is exploratory and what is not. I think nobody would ignore an interesting exploratory result. - archival value. Nobody knows whether the "success rate" of RR is higher equal or lower than traditional paper. Perhaps there are not many data yet. But we do know that current literature is inflated with false positive (e.g., look at the various replication experiments or at the analysis by Ioannidis et al.). In my opinion, the auditory field is a safe island (at least in comparison to other fields). However, I have no data about it. > Yes, we do "move more like a carpenter trying to adapt and adjust thing= s > in real time." Registered reports ask that we plan most, if not all, of > our measurements, cuts, and adjustments in advance. They are anathema t= o > the process. I would like to move from "carpenter" to "engineer" :-) RR and other standards that are now suggested (e.g., preregistration, Bayesian stats, multi-lab experiments) enable to do so (in my opinion). In any case, all journals that are adopting RR still offer the traditional submissions. So everything is preserved. > I agree with Nilesh's comments, especially, "Aren't we, as researchers, > possessed of sufficient integrity and ethics to present our research in > the correct light? If this core value is missing, I fear no external > policing is going to help." - I don't know. For example, here in Italy -where staff recruiting is screened by number of publications, H-index and number of citations- researchers are pushed a lot in the direction of "publish as much as possible in high impact factor journals and get cited a lot or perish". And in fact a recent editorial in Nature was highlighting that here in Italy the number of self-citations is increasing. I'm wandering whether others not-so-nice behaviours are also adopted (Interesting enough, Italy still scores zero (!) for number of scientific frauds. There was list in wikipedia, I can't find it, sorry.) If we look back at the history of psychology (my own field), it looks to me we are facing a change in scientific paradigm. From Wundt up to Titchener we were using (and trusting) introspection as a good tool to investigate psychology. Then we had the paper by John Watson (1913, Psychology as the behaviorist views it. Psychological review, 20(2), 158-177.) and in a few years time introspection was forgotten and neglected. Let's look back at Watson's statement now: it sounds so obvious today ("Introspection forms no essential part of its methods, nor is the scientific value of its data dependent upon the readiness with which they lend themselves to interpretation in terms of consciousness.). My guess is that in a few years time we will do the same for several of the current research practices. Apologies for the long email and ll the best from a scorchy hot Italy, m ps: no carpenter has been killed or injured while writing this email.