Subject: Re: [AUDITORY] stats use in psychology and hearing science From: James Johnston <audioskeptic@xxxxxxxx> Date: Sun, 23 Jun 2013 21:32:38 -0700 List-Archive:<http://lists.mcgill.ca/scripts/wa.exe?LIST=AUDITORY>--bcaec5489eeb04135e04dfdee75d Content-Type: text/plain; charset=ISO-8859-1 I have to agree with this take, but it's not recent. I have seen more than one or two subjective test that is effectively testing audibility (although not quite directly) that had a clear bimodal distribution (and for that I mean of "what was detected") that was stuffed through ANOVA with no attention to clustering at all. Mumble. On Sun, Jun 23, 2013 at 11:21 AM, Iftikhar Nizami <nizamii2@xxxxxxxx> wrote: > Dear List - My thanks to Holger Mitterer for pointing out the paper by > Simmons et al in Psych Sci, which promises to be an interesting read. It is > just one of a long string of papers in recent years which point out just > how little of value can arise through statistical testing of experimental > results (see also the numerous papers of John Ioannidis at Stanford on > this topic in medicine). > > Unfortunately, this problem of designing experiments for the data analysis > - and the wider problem of inappropriate experimental design and > inappropriate data analysis - is only going to get worse, especially in > departments of education, psychology, and hearing research. There, the > older generation of researchers, who might have had at least an undergrad > freshman calculus course, has been replaced by a new generation of workers > who do not have math beyond the 10th grade of high school and who barely > passed their weak undergraduate mandatory course in practical stats. Too > many people now seem to think of stats testing (ANOVA in particular) as an > act of magic that tells them what's "significant". It is exceedingly rare, > for example, to find any mention of whether the assumptions underlying the > statistical tests are actually obeyed, as no-one seems to realize that > statistical tests are derived from mathematical models that involve * > assumptions*. There is a solution to this problem: stricter math > requirements at the undergraduate and graduate levels, including > introductory *theoretical *statistics, not just basic stats testing. If > we're going to use stats, let's do it properly. - Lance Nizami PhD, Palo > Alto, Cal. > > *From:* Holger Mitterer <holgermitterer@xxxxxxxx> > *To:* AUDITORY@xxxxxxxx > *Sent:* Saturday, June 22, 2013 7:40 AM > *Subject:* [AUDITORY] Reminder: Speed Sound Finding Experiment > > probably many readers caught this, but just to make sure: > The reminder for the speech sound finding experiment contained a somewhat > questionable phrase: > > > We are missing a few participants to reach statistical significance so > > please consider giving it a try: > > Stopping data collection when an effect becomes significant is a very > problematic research strategy, > see the paper by Simmons et al. in PsychScience ( > http://pss.sagepub.com/content/22/11/1359.abstract). > > > > > -- James D. (jj) Johnston Independent Audio and Electroacoustics Consultant --bcaec5489eeb04135e04dfdee75d Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div>I have to agree with this take, but it's not rece= nt. I have seen more than one or two subjective test that is effectively te= sting audibility (although not quite directly) that had a clear bimodal dis= tribution (and for that I mean of "what was detected") that was s= tuffed through ANOVA with no attention to clustering at all.</div> <div>=A0</div><div>Mumble.</div></div><div class=3D"gmail_extra"><br><br><d= iv class=3D"gmail_quote">On Sun, Jun 23, 2013 at 11:21 AM, Iftikhar Nizami = <span dir=3D"ltr"><<a href=3D"mailto:nizamii2@xxxxxxxx" target=3D"_blank"= >nizamii2@xxxxxxxx</a>></span> wrote:<br> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"><div><div style=3D"font-family:times new rom= an,new york,times,serif;font-size:12pt"><div><span>Dear List - My thanks to= Holger Mitterer for pointing out the paper by Simmons et al in Psych Sci, = which promises to be an interesting read. It is just one of a long string o= f papers in recent years which point out just how little of value can arise= through statistical testing of experimental results (see also the numerous= pap<var></var>ers of John Ioannidis at Stanford on this topic in medicine)= .</span></div> <div><span></span>=A0</div> <div><span>Unfortunately, this problem of designing experiments for the dat= a analysis - and the wider problem of inappropriate experimental design and= inappropriate data analysis - is only going to get worse, especially in de= partments of education, psychology, and hearing research. There, the older = generation of researchers, who might have had at least an undergrad freshma= n calculus course, has been replaced by a new generation of workers who do = not have math beyond the 10th grade of high school and who barely passed th= eir weak undergraduate mandatory course in practical stats. Too many people= now seem to think of stats testing (ANOVA in particular) as an act of magi= c that tells them what's "significant". It is exceedingly rar= e, for example, to find any mention of whether the assumptions underlying t= he statistical tests are actually obeyed, as no-one seems to realize that s= tatistical tests are derived from mathematical models that involve <em>assumptions</em>.=A0There is a soluti= on to this problem: stricter math requirements at the undergraduate and gra= duate levels, including introductory <em>theoretical </em>statistics, not j= ust basic stats testing. If we're going to use stats, let's do it p= roperly. - Lance Nizami PhD, Palo Alto, Cal.</span></div> <div><br> <blockquote style=3D"padding-left:5px;margin-top:5px;margin-left:5px;border= -left-color:rgb(16,16,255);border-left-width:2px;border-left-style:solid"> <div style=3D"font-family:times new roman,new york,times,serif;font-size:12= pt"> <div style=3D"font-family:times new roman,new york,times,serif;font-size:12= pt"> <div dir=3D"ltr"> <div style=3D"margin:5px 0px;padding:0px;border:1px solid rgb(204,204,204);= line-height:0;font-size:0px;min-height:0px" readonly></div><font face=3D"Ar= ial"><b><span style=3D"font-weight:bold">From:</span></b> Holger Mitterer &= lt;<a href=3D"mailto:holgermitterer@xxxxxxxx" target=3D"_blank">holgermi= tterer@xxxxxxxx</a>><br> <b><span style=3D"font-weight:bold">To:</span></b> <a href=3D"mailto:AUDITO= RY@xxxxxxxx" target=3D"_blank">AUDITORY@xxxxxxxx</a> <br><b><= span style=3D"font-weight:bold">Sent:</span></b> Saturday, June 22, 2013 7:= 40 AM<br> <b><span style=3D"font-weight:bold">Subject:</span></b> [AUDITORY] Reminder= : Speed Sound Finding Experiment<br></font></div> <div><br> <div> <div> <div style=3D"font-family:tahoma,new york,times,serif;font-size:12pt"> <div>probably many readers caught this, but just to make sure:</div> <div>The reminder for the speech sound finding experiment contained a somew= hat questionable phrase:</div> <div style=3D"font-family:tahoma,new york,times,serif;font-size:16px;font-s= tyle:normal;background-color:transparent"><br></div> <div style=3D"font-family:tahoma,new york,times,serif;font-size:16px;font-s= tyle:normal;background-color:transparent">> We are missing a few partici= pants to reach statistical significance so<br>> please consider giving i= t a try:</div> <div style=3D"font-family:tahoma,new york,times,serif;font-size:16px;font-s= tyle:normal;background-color:transparent"><br></div> <div style=3D"font-family:tahoma,new york,times,serif;font-size:16px;font-s= tyle:normal;background-color:transparent">Stopping data collection when an = effect becomes significant is a very problematic research strategy,</div> <div style=3D"font-family:tahoma,new york,times,serif;font-size:16px;font-s= tyle:normal;background-color:transparent">see the paper by Simmons et al. i= n PsychScience (<a href=3D"http://pss.sagepub.com/content/22/11/1359.abstra= ct" target=3D"_blank">http://pss.sagepub.com/content/22/11/1359.abstract</a= >).</div> <div style=3D"font-family:tahoma,new york,times,serif;font-size:16px;font-s= tyle:normal;background-color:transparent"><br></div> <div style=3D"font-family:tahoma,new york,times,serif;font-size:16px;font-s= tyle:normal;background-color:transparent"><br></div></div></div></div><br><= br></div></div></div></blockquote></div></div></div></blockquote></div><br> <br clear=3D"all"><br>-- <br><div>James D. (jj) Johnston</div><div>Independ= ent Audio and Electroacoustics Consultant</div> </div> --bcaec5489eeb04135e04dfdee75d--