Subject: Re: [AUDITORY] Software for internet-based auditory testing From: "Richard F. Lyon" <dicklyon@xxxxxxxx> Date: Tue, 3 Oct 2017 21:38:26 -0700 List-Archive:<http://lists.mcgill.ca/scripts/wa.exe?LIST=AUDITORY>--94eb2c19568ec8b56a055ab12d91 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Many thanks, Sam and Bryan and Kevin and all those who replied privately. I can see many possible ways forward; just need to get pecking at some... Dick On Tue, Oct 3, 2017 at 7:06 PM, kevin woods <kevinwoods@xxxxxxxx> wrote: > Further to Sam's email, here is a link to a code package we put together > to implement our headphone screening task (intended to improve the qualit= y > of crowdsourced data): http://mcdermottlab.mit.edu/downloads.html > > We have generally found that the quality of data obtained online with our > screening procedure is comparable to that of data obtained in the lab on > the same experiments. For obvious reasons we have only run experiments > where precise stimulus control seems unlikely to be critical. > > Please feel free to contact us at kwoods@xxxxxxxx with questions. > > Sincerely, > > Kevin Woods (on behalf of the McDermott Lab, Department of Brain and > Cognitive Sciences, MIT) > > > On Tue, Oct 3, 2017 at 12:59 AM, Samuel Mehr <sam@xxxxxxxx> wrote: > >> Dear Dick, >> >> Lots of folks do successful audio-based experiments on Turk and I >> generally find it to be a good platform for the sort of work you're >> describing (which is not really what I do, but experimentally is similar >> enough for the purposes of your question). I've done a few simple listen= ing >> experiments of the form "listen to this thing, answer some questions abo= ut >> it", and the results directly replicate parallel in-person experiments i= n >> my lab, even when Turkers geolocate to lots of far-flung countries. I >> require subjects to wear headphones and validate that requirement with t= his >> great task from Josh McDermott's lab: >> >> Woods, K. J. P., Siegel, M. H., Traer, J., & McDermott, J. H. (2017). >> Headphone screening to facilitate web-based auditory experiments. *Atten= tion, >> Perception, & Psychophysics*, 1=E2=80=939. https://doi.org/10.3758/s1341= 4 >> -017-1361-2 >> <https://urldefense.proofpoint.com/v2/url?u=3Dhttps-3A__doi.org_10.3758_= s13414-2D017-2D1361-2D2&d=3DDwMFaQ&c=3DWO-RGvefibhHBZq3fL85hQ&r=3DhClPLRw3t= DEOn9yIUk8lXthixMA_5Xmkz_VcyZe7Nis&m=3D33Mv2SLQ46aahHEyH5V-aoAK1FI9yxgj--fQ= jCiOfBg&s=3DEMJagVRicwRwOoDLHL7J_KOVCZaPaWuy6gr0vYRmTK4&e=3D> >> >> In a bunch of piloting, passing the headphone screener correlates with a >> bunch of other checks on Turker compliance, positively. Things like "Wha= t >> color is the sky? Please answer incorrectly, on purpose" and "Tell us >> honestly how carefully you completed this HIT". Basically, if you have a >> few metrics in an experiment that capture variance on some dimension >> related to participant quality, you should be able to easily tell which >> Turkers are actually doing good work and which aren't. Depending on how >> your ethics approval is set up, you can either pay everyone and filter o= ut >> bad subjects, or require them to pass some level of quality control to >> receive payment. >> >> best >> Sam >> >> >> -- >> Samuel Mehr >> Department of Psychology >> Harvard University >> themusiclab.org >> <https://urldefense.proofpoint.com/v2/url?u=3Dhttp-3A__themusiclab.org_&= d=3DDwMFaQ&c=3DWO-RGvefibhHBZq3fL85hQ&r=3DhClPLRw3tDEOn9yIUk8lXthixMA_5Xmkz= _VcyZe7Nis&m=3D33Mv2SLQ46aahHEyH5V-aoAK1FI9yxgj--fQjCiOfBg&s=3DpCUtydTJaz47= M50EeAcCAChXuyOuAISrqupQZFMpbQ4&e=3D> >> naturalhistoryofsong.org >> <https://urldefense.proofpoint.com/v2/url?u=3Dhttp-3A__naturalhistoryofs= ong.org_&d=3DDwMFaQ&c=3DWO-RGvefibhHBZq3fL85hQ&r=3DhClPLRw3tDEOn9yIUk8lXthi= xMA_5Xmkz_VcyZe7Nis&m=3D33Mv2SLQ46aahHEyH5V-aoAK1FI9yxgj--fQjCiOfBg&s=3Dv6D= XeHQU7koDz4xM-awDfZV_oPoAjPtmyikYNxdtgH0&e=3D> >> >> >> >> On Tue, Oct 3, 2017 at 8:57 AM, Richard F. Lyon <dicklyon@xxxxxxxx> wrote= : >> >>> Five years on, are there any updates on experience using Mechanical Tur= k >>> and such for sound perception experiments? >>> >>> I've never conducted psychoacoustic experiments myself (other than >>> informal ones on myself), but now I think I have some modeling ideas th= at >>> need to be tuned and tested with corresponding experimental data. Is M= Turk >>> the way to go? If it is, are IRB approvals still needed? I don't even = know >>> if that applies to me; probably my company has corresponding approval >>> requirements. >>> >>> I'm interested in things like SNR thresholds for binaural detection and >>> localization of different types of signals and noises -- 2AFC tests who= se >>> relative results across conditions would hopefully not be strongly >>> dependent on level or headphone quality. Are there good MTurk task >>> structures that motivate people to do a good job on these, e.g. by maki= ng >>> their space quieter, paying attention, getting more pay as the task get= s >>> harder, or just getting to do more similar tasks, etc.? Can the pay de= pend >>> on performance? Or just cut them off when the SNR has been lowered to >>> threshold, so that people with lower thresholds stay on and get paid lo= nger? >>> >>> If anyone in academia has a good setup for human experiments and an >>> interest in collaborating on binaural model improvements, I'd love to >>> discuss that, too, either privately or on the list. >>> >>> Dick >>> >>> >>> --94eb2c19568ec8b56a055ab12d91 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr">Many thanks, Sam and Bryan and Kevin and all those who rep= lied privately.<br> <br> I can see many possible ways forward; just need to get pecking at some...<b= r> <br> Dick<div class=3D"gmail_extra"><br><div class=3D"gmail_quote">On Tue, Oct 3= , 2017 at 7:06 PM, kevin woods <span dir=3D"ltr"><<a href=3D"mailto:kevi= nwoods@xxxxxxxx" target=3D"_blank">kevinwoods@xxxxxxxx</a>>= ;</span> wrote:<br><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 = .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir=3D"ltr"><div sty= le=3D"font-size:12.8px">Further to Sam's email, here is a link to a cod= e package we put together to implement our headphone screening task (intend= ed to improve the quality of crowdsourced data):=C2=A0<a href=3D"http://mcd= ermottlab.mit.edu/downloads.html" target=3D"_blank">http://mcdermottlab.mit= <wbr>.edu/downloads.html</a></div><div style=3D"font-size:12.8px"><br></div= ><div style=3D"font-size:12.8px">We have generally found that the quality o= f data obtained online with our screening procedure is comparable to that o= f data obtained in the lab on the same experiments. For obvious reasons we = have only run experiments where precise stimulus control seems unlikely to = be critical.=C2=A0</div><span class=3D"m_6977978009752606249gmail-im" style= =3D"font-size:12.8px"><div><br></div><div>Please feel free to contact us at= =C2=A0<a href=3D"mailto:kwoods@xxxxxxxx" target=3D"_blank">kwoods@xxxxxxxx</a= >=C2=A0with questions.</div><div><br></div><div>Sincerely,</div><div><br></= div></span><div style=3D"font-size:12.8px">Kevin Woods (on behalf of the Mc= Dermott Lab, Department of Brain and Cognitive Sciences, MIT)</div><div cla= ss=3D"m_6977978009752606249gmail-yj6qo m_6977978009752606249gmail-ajU" styl= e=3D"margin:2px 0px 0px;font-size:12.8px"><div id=3D"m_6977978009752606249g= mail-:jp" class=3D"m_6977978009752606249gmail-ajR"><img class=3D"m_69779780= 09752606249gmail-ajT" src=3D"https://ssl.gstatic.com/ui/v1/icons/mail/image= s/cleardot.gif"></div><div id=3D"m_6977978009752606249gmail-:jp" class=3D"m= _6977978009752606249gmail-ajR"><br></div></div><div class=3D"gmail_extra"><= br><div class=3D"gmail_quote"><span class=3D"">On Tue, Oct 3, 2017 at 12:59= AM, Samuel Mehr <span dir=3D"ltr"><<a href=3D"mailto:sam@xxxxxxxx= u" target=3D"_blank">sam@xxxxxxxx</a>></span> wrote:<br></span><b= lockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-le= ft:1px solid rgb(204,204,204);padding-left:1ex"><div dir=3D"ltr">Dear Dick,= <div><br></div><div><span class=3D"">Lots of folks do successful audio-base= d experiments on Turk and I generally find it to be a good platform for the= sort of work you're describing (which is not really what I do, but exp= erimentally is similar enough for the purposes of your question). I've = done a few simple listening experiments of the form "listen to this th= ing, answer some questions about it", and the results directly replica= te parallel in-person experiments in my lab, even when Turkers geolocate to= lots of far-flung countries. I require subjects to wear headphones and val= idate that requirement with this great task from Josh McDermott's lab:<= div><span><br></span></div><div><span>Woods, K. J. P., Siegel, M. H., Traer= , J., & McDermott, J. H. (2017). Headphone screening to facilitate web-= based auditory experiments. </span><i>Attention, Perception, & Psychoph= ysics</i><span>, 1=E2=80=939. <a href=3D"https://urldefense.proofpoint.com/= v2/url?u=3Dhttps-3A__doi.org_10.3758_s13414-2D017-2D1361-2D2&d=3DDwMFaQ= &c=3DWO-RGvefibhHBZq3fL85hQ&r=3DhClPLRw3tDEOn9yIUk8lXthixMA_5Xmkz_V= cyZe7Nis&m=3D33Mv2SLQ46aahHEyH5V-aoAK1FI9yxgj--fQjCiOfBg&s=3DEMJagV= RicwRwOoDLHL7J_KOVCZaPaWuy6gr0vYRmTK4&e=3D" target=3D"_blank">https://d= oi.org/10.3758/s13414<wbr>-017-1361-2</a></span><div class=3D"m_69779780097= 52606249gmail-m_6944949973339543626m_-6026670406689544790m_-335280008535405= 6723m_-6318531832382596260gmail-m_-3941813762889735983gmail-csl-bib-body" s= tyle=3D"line-height:2;margin-left:2em"> =20 <span class=3D"m_6977978009752606249gmail-m_6944949973339543626m_-6026670= 406689544790m_-3352800085354056723m_-6318531832382596260gmail-m_-3941813762= 889735983gmail-Z3988" title=3D"url_ver=3DZ39.88-2004&ctx_ver=3DZ39.88-2= 004&rfr_id=3Dinfo%3Asid%2Fzotero.org%3A2&rft_id=3Dinfo%3Adoi%2F10.3= 758%2Fs13414-017-1361-2&rft_val_fmt=3Dinfo%3Aofi%2Ffmt%3Akev%3Amtx%3Ajo= urnal&rft.genre=3Darticle&rft.atitle=3DHeadphone%20screening%20to%2= 0facilitate%20web-based%20auditory%20experiments&rft.jtitle=3DAttention= %2C%20Perception%2C%20%26%20Psychophysics&rft.stitle=3DAtten%20Percept%= 20Psychophys&rft.aufirst=3DKevin%20J.%20P.&rft.aulast=3DWoods&r= ft.au=3DKevin%20J.%20P.%20Woods&rft.au=3DMax%20H.%20Siegel&rft.au= =3DJames%20Traer&rft.au=3DJosh%20H.%20McDermott&rft.date=3D2017-07-= 10&rft.pages=3D1-9&rft.spage=3D1&rft.epage=3D9&rft.issn=3D1= 943-3921%2C%201943-393X&rft.language=3Den"></span></div></div><div clas= s=3D"gmail_extra"><br></div><div class=3D"gmail_extra">In a bunch of piloti= ng, passing the headphone screener correlates with a bunch of other checks = on Turker compliance, positively. Things like "What color is the sky? = Please answer incorrectly, on purpose" and "Tell us honestly how = carefully you completed this HIT". Basically, if you have a few metric= s in an experiment that capture variance on some dimension related to parti= cipant quality, you should be able to easily tell which Turkers are actuall= y doing good work and which aren't. Depending on how your ethics approv= al is set up, you can either pay everyone and filter out bad subjects, or r= equire them to pass some level of quality control to receive payment.</div>= <div class=3D"gmail_extra"><br></div><div class=3D"gmail_extra">best</div><= div class=3D"gmail_extra">Sam</div><span class=3D"m_6977978009752606249gmai= l-m_6944949973339543626m_-6026670406689544790HOEnZb"><font color=3D"#888888= "><div class=3D"gmail_extra"><div><br class=3D"m_6977978009752606249gmail-m= _6944949973339543626m_-6026670406689544790m_-3352800085354056723m_-63185318= 32382596260gmail-Apple-interchange-newline"><br></div>--=C2=A0<br><div clas= s=3D"m_6977978009752606249gmail-m_6944949973339543626m_-6026670406689544790= m_-3352800085354056723m_-6318531832382596260gmail-m_-3941813762889735983gma= il_signature"><div dir=3D"ltr"><div dir=3D"ltr"><div dir=3D"ltr"><div dir= =3D"ltr"><div dir=3D"ltr"><div dir=3D"ltr"><div dir=3D"ltr"><div dir=3D"ltr= "><div dir=3D"ltr"><div dir=3D"ltr"><div dir=3D"ltr"><div dir=3D"ltr"><div = dir=3D"ltr"><div dir=3D"ltr"><div dir=3D"ltr"><div dir=3D"ltr"><div dir=3D"= ltr"><div dir=3D"ltr"><div dir=3D"ltr"><div dir=3D"ltr"><div dir=3D"ltr"><d= iv style=3D"font-size:12.8px"><div style=3D"font-size:12.8px"><div style=3D= "font-size:12.8px"><div style=3D"font-size:12.8px"><div style=3D"font-size:= 12.8px"><div style=3D"font-size:12.8px"><div style=3D"font-size:12.8px"><di= v style=3D"color:rgb(136,136,136);font-size:12.8px"><div dir=3D"ltr" style= =3D"color:rgb(34,34,34);font-size:12.8px"><div dir=3D"ltr" style=3D"font-si= ze:12.8px"><div dir=3D"ltr" style=3D"font-size:12.8px"><div dir=3D"ltr" sty= le=3D"font-size:12.8px"><div dir=3D"ltr" style=3D"font-size:12.8px"><div di= r=3D"ltr" style=3D"font-size:12.8px"><font color=3D"#666666">Samuel Mehr</f= ont></div><div style=3D"font-size:12.8px"><font color=3D"#666666">Departmen= t of Psychology</font></div><div dir=3D"ltr" style=3D"font-size:12.8px"><sp= an style=3D"color:rgb(102,102,102);font-size:12.8px">Harvard University</sp= an><br></div><div dir=3D"ltr" style=3D"font-size:12.8px"><font color=3D"#66= 6666"><a href=3D"https://urldefense.proofpoint.com/v2/url?u=3Dhttp-3A__them= usiclab.org_&d=3DDwMFaQ&c=3DWO-RGvefibhHBZq3fL85hQ&r=3DhClPLRw3= tDEOn9yIUk8lXthixMA_5Xmkz_VcyZe7Nis&m=3D33Mv2SLQ46aahHEyH5V-aoAK1FI9yxg= j--fQjCiOfBg&s=3DpCUtydTJaz47M50EeAcCAChXuyOuAISrqupQZFMpbQ4&e=3D" = target=3D"_blank">themusiclab.org</a></font></div><div dir=3D"ltr" style=3D= "font-size:12.8px"><font color=3D"#666666"><a href=3D"https://urldefense.pr= oofpoint.com/v2/url?u=3Dhttp-3A__naturalhistoryofsong.org_&d=3DDwMFaQ&a= mp;c=3DWO-RGvefibhHBZq3fL85hQ&r=3DhClPLRw3tDEOn9yIUk8lXthixMA_5Xmkz_Vcy= Ze7Nis&m=3D33Mv2SLQ46aahHEyH5V-aoAK1FI9yxgj--fQjCiOfBg&s=3Dv6DXeHQU= 7koDz4xM-awDfZV_oPoAjPtmyikYNxdtgH0&e=3D" style=3D"font-size:12.8px" ta= rget=3D"_blank">naturalhistoryofsong.org</a></font></div></div></div></div>= </div></div></div></div></div></div></div></div></div></div></div></div></d= iv></div></div></div></div></div></div></div></div></div></div></div></div>= </div></div></div></div></div></div></div></div></font></span></span><div><= div class=3D"m_6977978009752606249gmail-m_6944949973339543626m_-60266704066= 89544790h5"><div class=3D"gmail_extra"><br></div><div class=3D"gmail_extra"= ><br></div><div class=3D"gmail_extra"><br><div class=3D"gmail_quote"><span = class=3D"">On Tue, Oct 3, 2017 at 8:57 AM, Richard F. Lyon <span dir=3D"ltr= "><<a href=3D"mailto:dicklyon@xxxxxxxx" target=3D"_blank">dicklyon@xxxxxxxx= g</a>></span> wrote:<br></span><blockquote class=3D"gmail_quote" style= =3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding= -left:1ex"><span class=3D""><div dir=3D"ltr"><div><div><div>Five years on, = are there any updates on experience using Mechanical Turk and such for soun= d perception experiments?<br><br></div>I've never conducted psychoacous= tic experiments myself (other than informal ones on myself), but now I thin= k I have some modeling ideas that need to be tuned and tested with correspo= nding experimental data.=C2=A0 Is MTurk the way to go?=C2=A0 If it is, are = IRB approvals still needed? I don't even know if that applies to me; pr= obably my company has corresponding approval requirements.<br><br></div>I&#= 39;m interested in things like SNR thresholds for binaural detection and lo= calization of different types of signals and noises -- 2AFC tests whose rel= ative results across conditions would hopefully not be strongly dependent o= n level or headphone quality.=C2=A0 Are there good MTurk task structures th= at motivate people to do a good job on these, e.g. by making their space qu= ieter, paying attention, getting more pay as the task gets harder, or just = getting to do more similar tasks, etc.?=C2=A0 Can the pay depend on perform= ance?=C2=A0 Or just cut them off when the SNR has been lowered to threshold= , so that people with lower thresholds stay on and get paid longer?</div><d= iv><br></div><div>If anyone in academia has a good setup for human experime= nts and an interest in collaborating on binaural model improvements, I'= d love to discuss that, too, either privately or on the list.<br></div><div= ><br></div>Dick<br><div><br></div></div></span><br></blockquote></div></div= ></div></div></div></div></blockquote></div></div></div></blockquote></div>= </div></div> --94eb2c19568ec8b56a055ab12d91--