Subject: [AUDITORY] Registered reports From: Nilesh Madhu <000000405df1884c-dmarc-request@xxxxxxxx> Date: Tue, 5 Jun 2018 13:16:17 +0200 List-Archive:<http://lists.mcgill.ca/scripts/wa.exe?LIST=AUDITORY>Dear Tim, I appreciate your initiative towards reproducible research. However I fear that registered reports would just add another layer of overhead to academics and students already under the pressure to publish. If I understand correctly, this involves two rounds of review: a first review based on the methodology and evaluation and a second based on the results of the research. For each stage, probably at least two review rounds would be needed (going by the current publishing cycle). I fear, as Gaston does, this might stifle creativity and lead to overwork also for reviewers and editors. Of course, this is assuming you want to make registered reports compulsory... Furthermore, such an approach may not be equally applicable to all research. For research into algorithms, for example, the value of the research lies, usually, in the core idea. There are myriad accepted forms of evaluation and to force a strict evaluation pattern/methodology would be counterproductive. Reproducible research in this case is targeted by encouraging authors to make their code and test data public. What I would support are (voluntary) guidelines on reporting results of experiments. This is often to be found in in the engineering field, when one participates in an open challenge. Lastly, the main reason for this initiative is to avoid 'mis-reporting' the results in favour of a hypothesis. Surely, this calls for self policing? Aren't we, as researchers, possessed of sufficient integrity and ethics to present our research in the correct light? If this core value is missing, I fear no external policing is going to help. Best regards Nilesh Madhu