Subject: Re: [AUDITORY] Registered reports From: Les Bernstein <lbernstein@xxxxxxxx> Date: Wed, 6 Jun 2018 14:20:41 -0400 List-Archive:<http://lists.mcgill.ca/scripts/wa.exe?LIST=AUDITORY>--------------7F037E0875767145BF778EA0 Content-Type: text/html; charset="windows-1252" Content-Transfer-Encoding: quoted-printable X-MIME-Autoconverted: from 8bit to quoted-printable by edgeum2.it.mcgill.ca id w56ILaPY030363 <html> <head> <meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Dwindows-1252"> </head> <body text=3D"#000000" bgcolor=3D"#FFFFFF"> <font size=3D"-1"><font face=3D"Verdana">Well, we simply don't agree.= =A0 Separate RRs for each experiment when one is a follow-up or control based on the first?=A0 No, I see no advantage there and many disadvantages.=A0 I think the distinction between "confirmatory" and "exploratory" analyses, as described, is artificial and without merit.=A0 Performing an unanticipated statistical (or other) analysis because it is warranted by the data is not "exploratory."<br> <br> I don't think we will ever rid ourselves of all aspects of being carpenters.=A0 Much as many do not wish to admit it, there is art involved in doing science.=A0 I don't mean that in the sense of subjectivity but in the sense of having intuitive knowledge and the judgment to define the path to be taken.=A0 I believe that th= e POSITIVE history of science supports such a view.=A0 I hope we ar= e not facing the change in paradigm you herald.=A0 I think much of value might be lost if we were.<br> <br> There's nothing new under the sun.=A0 Generations have faced the notion of "publish or perish" in academia.=A0 As I see it, imposing stilted, restricted rules in the name of objectivity that, in and of themselves, have the potential to degrade scientific progress is not the answer.=A0 Again, the intent is laudable, but in my view, the proposed solution is fundamentally flawed.=A0 <br> <br> Thanks for the history-- my field is Experimental Psychology.=A0 = I know well the stories. <span class=3D"moz-smiley-s1"><span>:-)</s= pan></span><br> </font></font><br> <div class=3D"moz-cite-prefix"><font size=3D"-1" face=3D"Verdana">Bes= t from the cloudy and cool east coast of the US.<br> <br> Les<br> </font><br> On 6/6/2018 12:15 PM, Massimo Grassi wrote:<br> </div> <blockquote type=3D"cite" cite=3D"mid:85d674fd-e12b-4e09-f5a2-60025213676a@xxxxxxxx">Les, <br> <br> <blockquote type=3D"cite">Thanks for your response.=A0 Note that I mentioned a number of issues that <br> I identify as problems and shortcomings and not just a single one.=A0 That <br> the "results" section of a Stage 2 submission allows for "exploratory <br> analysis" hardly addresses the issues I raised with regard to <br> hypotheses, follow-up and control experiments, choice of PRIMARY <br> statistical tests, and archival value.=A0 Furthermore, the "exploratory" <br> analyses you cite are clearly considered subordinate to the pre-approved <br> "confirmatory analyses."=A0 See Nosek and Lakens (2014).=A0 As I = see it, <br> that's unnecessarily restrictive. <br> </blockquote> <br> - Multiple experiments. Usually, Experiment 2 follows the results of <br> Experiment 1 (that in RR are unknown). One solution is that you do a <br> registered report of Experiment 1 and a new registered report for <br> Experiment 2 <br> - confirmatory vs exploratory analysis. This is exactly the point. <br> Nowadays often we sell "exploratory" like "confirmatory". In contrast, <br> we should make clear what is exploratory and what is not. I think nobody <br> would ignore an interesting exploratory result. <br> - archival value. Nobody knows whether the "success rate" of RR is <br> higher equal or lower than traditional paper. Perhaps there are not many <br> data yet. But we do know that current literature is inflated with false <br> positive (e.g., look at the various replication experiments or at the <br> analysis by Ioannidis et al.). In my opinion, the auditory field is a <br> safe island (at least in comparison to other fields). However, I have no <br> data about it. <br> <br> <blockquote type=3D"cite">Yes, we do "move more like a carpenter trying to adapt and adjust things <br> in real time." Registered reports ask that we plan most, if not all, of <br> our measurements, cuts, and adjustments in advance. They are anathema to <br> the process. <br> </blockquote> <br> I would like to move from "carpenter" to "engineer" :-) <br> RR and other standards that are now suggested (e.g., preregistration, <br> Bayesian stats, multi-lab experiments) enable to do so (in my opinion). <br> In any case, all journals that are adopting RR still offer the <br> traditional submissions. So everything is preserved. <br> <br> <blockquote type=3D"cite">I agree with Nilesh's comments, especially, "Aren't we, as researchers, <br> possessed of sufficient integrity and ethics to present our research in <br> the correct light? If this core value is missing, I fear no external <br> policing is going to help." <br> </blockquote> <br> - I don't know. For example, here in Italy -where staff recruiting is <br> screened by number of publications, H-index and number of citations- <br> researchers are pushed a lot in the direction of "publish as much as <br> possible in high impact factor journals and get cited a lot or perish". <br> And in fact a recent editorial in Nature was highlighting that here in <br> Italy the number of self-citations is increasing. I'm wandering whether <br> others not-so-nice behaviours are also adopted (Interesting enough, <br> Italy still scores zero (!) for number of scientific frauds. There was <br> list in wikipedia, I can't find it, sorry.) <br> <br> If we look back at the history of psychology (my own field), it looks to <br> me we are facing a change in scientific paradigm. From Wundt up to <br> Titchener we were using (and trusting) introspection as a good tool to <br> investigate psychology. Then we had the paper by John Watson (1913, <br> Psychology as the behaviorist views it. Psychological review, 20(2), <br> 158-177.) and in a few years time introspection was forgotten and <br> neglected. Let's look back at Watson's statement now: it sounds so <br> obvious today ("Introspection forms no essential part of its methods, <br> nor is the scientific value of its data dependent upon the readiness <br> with which they lend themselves to interpretation in terms of <br> consciousness.). My guess is that in a few years time we will do the <br> same for several of the current research practices. <br> <br> Apologies for the long email and ll the best from a scorchy hot Italy, <br> <br> m <br> <br> ps: no carpenter has been killed or injured while writing this email. <br> <br> </blockquote> <br> <div class=3D"moz-signature">-- <br> <meta http-equiv=3D"content-type" content=3D"text/html; charset=3Dwindows-1252"> <title></title> <b><span style=3D"font-size:10.0pt;font-family:"Arial",sans-se= rif; mso-fareast-font-family:"Times New Roman \;color\:\#002048"">Leslie R. Bernstein, Ph.D. </span></b>= <b><span style=3D"font-size:10.0pt;font-family:"Arial",sans-serif; mso-fareast-font-family:"Times New Roman \;color\:\#585858"">| </span></b><span style=3D"font-size:10.0pt;font-family:"Arial",sans-serif;mso-fa= reast-font-family:"Times New Roman \;color\:\#585858"">Professor</span><span style=3D"mso-fareast-font-family: "Times New Roman""><o= :p></o:p></span><span style=3D"font-size:10.0pt;font-family:"Arial",sans-serif; mso-fareast-font-family:"Times New Roman \;color\:\#585858""><br> Depts. of Neuroscience and Surgery (Otolaryngology)| UConn School of Medicine </span><br> <span style=3D"font-size:10.0pt;font-family:"Arial",sans-seri= f; mso-fareast-font-family:"Times New Roman \;color\:\#585858""></span><span style=3D"font-size:10.0pt;font-family:"Arial",sans-seri= f; mso-fareast-font-family:"Times New Roman \;color\:\#585858"">263 Farmington Avenue, Farmington, CT 06030-3401</span><br> <span style=3D"font-size:10.0pt;font-family:"Arial",sans-seri= f; mso-fareast-font-family:"Times New Roman \;color\:\#585858""></span><span style=3D"font-size:10.0pt;font-family:"Arial",sans-seri= f; mso-fareast-font-family:"Times New Roman \;color\:\#585858"">Office: 860.679.4622 | Fax: 860.679.2495<br> <br> <img alt=3D"" src=3D"cid:part1.4694CC9B.1A95D2FA@xxxxxxxx" height=3D"48" width=3D"125"><br> </span> </div> </body> </html> --------------7F037E0875767145BF778EA0 Content-Type: image/png; name="uconnhealth_stacked_blue_email.png" Content-ID: <part1.4694CC9B.1A95D2FA@xxxxxxxx> Content-Disposition: inline; filename="uconnhealth_stacked_blue_email.png" Content-Transfer-Encoding: base64 iVBORw0KGgoAAAANSUhEUgAAAH0AAAAwCAMAAAALmIWlAAAAGXRFWHRTb2Z0d2FyZQBBZG9i ZSBJbWFnZVJlYWR5ccllPAAAADNQTFRFKzVYHCZM4eLn8PHzpKi30dTbaG+IlZqrs7fDWWB8 O0Nkd32UwsXPSlJwhoufDRhA////A68jmAAAABF0Uk5T/////////////////////wAlrZli AAACYklEQVR42uzY22KDIAwAUC7e6gjw/187gUC4OetaupflaTXKWSuBKNMuJptC+QMCP20a uPHBQCubBWU46K24uB7NfZ58ZqaMP8D8AEBHtT/goXnB8WPwBf8r1WYUXWxUOZobHcJXIMcf ONclmE4s0lrRzYBIOjvTjX5WV6YbP2ZQJ6TWjfiAbuYzHW7p+xpuolQrP83sjQ5nupnu6Lqa J92MbnSznulcfkBHpNXNY6y+c0JKnWX1eFeflQt5qcNESKnrQMrf6LSAXeiWkEoXdOU4nZBK xzPnoTohtW5ZZyK8W09Io6tYjyP1iDS6XbAeh815S0irS6zHUfXuB0Ok1S3W41AdEWh1/Nb7 UD1bWytd0LY4ThdnerYbBJ1taT5XBmy4J8mNnWagp5fbba6HeiTdneK6vNgoFj0EhyO4eSZT 6AlpdNXoNzqYi94GbIU0eqhHp3d5drRf4heZUo9Iq8vYptup6VB37H6m/W6m0hFpdaxH3/yK 9cHpVq5Zy59n2GMV/QxPGdV9NElPE9njyOozsfWWKq6j7bLuw97KPBnM/mX86//6n+tH/yLK P7AUXcyd80Lx53WnspB47ty5qKND1gLockfIFzHR7srpQ7X0+6GgHPNFXedPyaN1CPFli71z +ZBeTxbcI2RXxwOQj/lW/Ui610Z6gA4a32WRrssXYbO76V/Zy6FLnZVj/qCnqGcdZE1DeFKc ntWrMV/RZXCh7B/fpF/+8tQQx6J74y9/Oeuyrm75+Jyf8oKWH9CL1SYNJtJ91HTOAD2fdjPN 9UcsumwmDNYXqvPtOLa9pH8LMACnoV0siZAyOAAAAABJRU5ErkJggg== --------------7F037E0875767145BF778EA0--