Subject: Re: [AUDITORY] stats (mis)use in psychology and hearing science From: Brian Gygi <bgygi@xxxxxxxx> Date: Mon, 24 Jun 2013 07:04:47 +0000 List-Archive:<http://lists.mcgill.ca/scripts/wa.exe?LIST=AUDITORY>----=_vm_0011_W5681024809_12147_1372057487 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable The problem is not just with higher education. As long as one needs a sig= nificant effect to get published, people are going to continue using the = easiest and most powerfult stats tests out there. Reviewers are complicit= in this as well - I have rarely seen a paper turned down because of inap= propriate statistical tests (even though many publications specifically a= sk about this). We as scientists could start by cleaning up our own shop = a bit. Brian Gygi, Ph.D. -----Original Message----- From: Iftikhar Nizami [mailto:nizamii2@xxxxxxxx Sent: Sunday, June 23, 2013 11:21 AM To: AUDITORY@xxxxxxxx Subject: Re: stats use in psychology and hearing science Dear List - My thanks to Holger Mitterer for pointing out the paper by Si= mmons et al in Psych Sci, which promises to be an interesting read. It is= just one of a long string of papers in recent years which point out just= how little of value can arise through statistical testing of experimenta= l results (see also the numerous papers of John Ioannidis at Stanford on = this topic in medicine). Unfortunately, this problem of designing experiments for the data analysi= s - and the wider problem of inappropriate experimental design and inappr= opriate data analysis - is only going to get worse, especially in departm= ents of education, psychology, and hearing research. There, the older gen= eration of researchers, who might have had at least an undergrad freshman= calculus course, has been replaced by a new generation of workers who do= not have math beyond the 10th grade of high school and who barely passed= their weak undergraduate mandatory course in practical stats. Too many p= eople now seem to think of stats testing (ANOVA in particular) as an act = of magic that tells them what's "significant". It is exceedingly rare, fo= r example, to find any mention of whether the assumptions underlying the = statistical tests are actually obeyed, as no-one seems to realize that st= atistical tests are derived from mathematical models that involve assumpt= ions. There is a solution to this problem: stricter math requirements at = the undergraduate and graduate levels, including introductory theoretical= statistics, not just basic stats testing. If we're going to use stats, l= et's do it properly. - Lance Nizami PhD, Palo Alto, Cal. From: Holger Mitterer <holgermitterer@xxxxxxxx> To: AUDITORY@xxxxxxxx Sent: Saturday, June 22, 2013 7:40 AM Subject: [AUDITORY] Reminder: Speed Sound Finding Experiment probably many readers caught this, but just to make sure: The reminder for the speech sound finding experiment contained a somewhat= questionable phrase: > We are missing a few participants to reach statistical significance so > please consider giving it a try: Stopping data collection when an effect becomes significant is a very pro= blematic research strategy, see the paper by Simmons et al. in PsychScience (http://pss.sagepub.com/c= ontent/22/11/1359.abstract). ----=_vm_0011_W5681024809_12147_1372057487 Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable <html><div><font face=3D"Verdana" size=3D"2"> </font></div> The problem is not just with higher education. As long as one needs= a significant effect to get published, people are going to continue usin= g the easiest and most powerfult stats tests out there. Reviewers a= re complicit in this as well - I have rarely seen a paper turned down bec= ause of inappropriate statistical tests (even though many publications sp= ecifically ask about this). We as scientists could start by cleanin= g up our own shop a bit.<br><br>Brian Gygi, Ph.D.<div><font color=3D"#000= 0ff" face=3D"Verdana" size=3D"2"></font> </div> <blockquote style=3D"PADDING-LEFT: 5px; MARGIN-LEFT: 5px; BORDER-LEFT: #0= 000ff 2px solid; MARGIN-RIGHT: 0px"><font face=3D"Tahoma" size=3D"2">----= -Original Message-----<br><b>From:</b> Iftikhar Nizami [mailto:nizamii2@xxxxxxxx= TT.NET]<br><b>Sent:</b> Sunday, June 23, 2013 11:21 AM<br><b>To:</b> AUDI= TORY@xxxxxxxx<br><b>Subject:</b> Re: stats use in psychology and h= earing science<br><br></font><table id=3D"201179" border=3D"0" height=3D"= 400" width=3D"100%"><tbody><tr><td valign=3D"top" width=3D"100%"><div sty= le=3D"color:#000; background-color:#fff; font-family:times new roman, new= york, times, serif;font-size:12pt"><div style=3D"RIGHT: auto"><span styl= e=3D"RIGHT: auto">Dear List - My thanks to Holger Mitterer for pointing o= ut the paper by Simmons et al in Psych Sci, which promises to be an inter= esting read. It is just one of a long string of papers in recent years wh= ich point out just how little of value can arise through statistical test= ing of experimental results (see also the numerous pap<var id=3D"yui-ie-c= ursor"></var>ers of John Ioannidis at Stanford on this topic in medicine)= .</span></div> <div style=3D"RIGHT: auto"><span style=3D"RIGHT: auto"></span> </div= > <div style=3D"RIGHT: auto"><span style=3D"RIGHT: auto">Unfortunately, thi= s problem of designing experiments for the data analysis - and the wider = problem of inappropriate experimental design and inappropriate data analy= sis - is only going to get worse, especially in departments of education,= psychology, and hearing research. There, the older generation of researc= hers, who might have had at least an undergrad freshman calculus course, = has been replaced by a new generation of workers who do not have math bey= ond the 10th grade of high school and who barely passed their weak underg= raduate mandatory course in practical stats. Too many people now seem to = think of stats testing (ANOVA in particular) as an act of magic that tell= s them what's "significant". It is exceedingly rare, for example, to find= any mention of whether the assumptions underlying the statistical tests = are actually obeyed, as no-one seems to realize that statistical tests ar= e derived from mathematical models that involve <i>assumptions</i>. There is a sol= ution to this problem: stricter math requirements at the undergraduate an= d graduate levels, including introductory <i style=3D"RIGHT: auto">theore= tical </i>statistics, not just basic stats testing. If we're going to use= stats, let's do it properly. - Lance Nizami PhD, Palo Alto, Cal.</span><= /div> <div style=3D"RIGHT: auto"><br style=3D"RIGHT: auto"> <blockquote style=3D"BORDER-LEFT: rgb(16,16,255) 2px solid; MARGIN-TOP: 5= px; PADDING-LEFT: 5px; MARGIN-LEFT: 5px; RIGHT: auto"> <div style=3D"FONT-FAMILY: times new roman, new york, times, serif; FONT-= SIZE: 12pt"> <div style=3D"FONT-FAMILY: times new roman, new york, times, serif; FONT-= SIZE: 12pt"> <div style=3D"RIGHT: auto" dir=3D"ltr"> <div style=3D"BORDER-BOTTOM: #ccc 1px solid; BORDER-LEFT: #ccc 1px solid;= PADDING-BOTTOM: 0px; LINE-HEIGHT: 0; MARGIN: 5px 0px; PADDING-LEFT: 0px;= PADDING-RIGHT: 0px; HEIGHT: 0px; FONT-SIZE: 0px; BORDER-TOP: #ccc 1px so= lid; BORDER-RIGHT: #ccc 1px solid; PADDING-TOP: 0px" class=3D"hr" readonl= y=3D"true" contenteditable=3D"false"></div><font face=3D"Arial" size=3D"2= "><b><span style=3D"FONT-WEIGHT: bold">From:</span></b> Holger Mitterer &= lt;holgermitterer@xxxxxxxx><br><b><span style=3D"FONT-WEIGHT: bold"= >To:</span></b> AUDITORY@xxxxxxxx <br><b><span style=3D"FONT-WEIGH= T: bold">Sent:</span></b> Saturday, June 22, 2013 7:40 AM<br><b><span sty= le=3D"FONT-WEIGHT: bold">Subject:</span></b> [AUDITORY] Reminder: Speed S= ound Finding Experiment<br></font></div> <div style=3D"RIGHT: auto" class=3D"y_msg_container"><br> <div style=3D"RIGHT: auto" id=3D"yiv1008380456"> <div style=3D"RIGHT: auto"> <div style=3D"BACKGROUND-COLOR: #fff; FONT-FAMILY: tahoma, new york, time= s, serif; COLOR: #000; FONT-SIZE: 12pt"> <div>probably many readers caught this, but just to make sure:</div> <div>The reminder for the speech sound finding experiment contained a som= ewhat questionable phrase:</div> <div style=3D"BACKGROUND-COLOR: transparent; FONT-STYLE: normal; FONT-FAM= ILY: tahoma, new york, times, serif; COLOR: rgb(0,0,0); FONT-SIZE: 16px">= <br></div> <div style=3D"BACKGROUND-COLOR: transparent; FONT-STYLE: normal; FONT-FAM= ILY: tahoma, new york, times, serif; COLOR: rgb(0,0,0); FONT-SIZE: 16px">= > We are missing a few participants to reach statistical significance = so<br>> please consider giving it a try:</div> <div style=3D"BACKGROUND-COLOR: transparent; FONT-STYLE: normal; FONT-FAM= ILY: tahoma, new york, times, serif; COLOR: rgb(0,0,0); FONT-SIZE: 16px">= <br></div> <div style=3D"BACKGROUND-COLOR: transparent; FONT-STYLE: normal; FONT-FAM= ILY: tahoma, new york, times, serif; COLOR: rgb(0,0,0); FONT-SIZE: 16px">= Stopping data collection when an effect becomes significant is a very pro= blematic research strategy,</div> <div style=3D"BACKGROUND-COLOR: transparent; FONT-STYLE: normal; FONT-FAM= ILY: tahoma, new york, times, serif; COLOR: rgb(0,0,0); FONT-SIZE: 16px">= see the paper by Simmons et al. in PsychScience (http://pss.sagepub.com/c= ontent/22/11/1359.abstract).</div> <div style=3D"BACKGROUND-COLOR: transparent; FONT-STYLE: normal; FONT-FAM= ILY: tahoma, new york, times, serif; COLOR: rgb(0,0,0); FONT-SIZE: 16px">= <br></div> <div style=3D"BACKGROUND-COLOR: transparent; FONT-STYLE: normal; FONT-FAM= ILY: tahoma, new york, times, serif; COLOR: rgb(0,0,0); FONT-SIZE: 16px">= <br></div></div></div></div><br><br></div></div></div></blockquote></div>= </div></td></tr></tbody></table></blockquote></html> ----=_vm_0011_W5681024809_12147_1372057487--