Subject: Re: [AUDITORY] Semantic McGurk Effect From: Sharath Chandra Ram <looplogic@xxxxxxxx> Date: Sat, 8 Aug 2020 23:54:38 -0500--0000000000004f8d9505ac6aa38a Content-Type: text/plain; charset="UTF-8" It's ever so interesting, and I think we had a round on a similar illusory conjunction before, with the Laurel-Yanny conundrum. Incidentally, that same year was the *"BrainStorm" / "Green Needle"* conjunction : https://www.youtube.com/watch?v=1okD66RmktA In this case you heard what you want to hear! (Don't bet on it as some folk hear neither of it, or are biased to one or the other) - I replicated (with due attribution) and extended upon the Current Biology paper on Laurel_yanny by Pressnitzer & Jackson Graves (et al) if anyone is interested in this non-peer reviewed pre-print on SSRN, with interrupted speech perception and correlation with hearing loss. Ramakrishnan, Sharath Chandra, Illusory Conjunctions in Speech Perception as a Lens into the Characteristics of Hearing Loss (August 24, 2019). Available at SSRN: https://ssrn.com/abstract=3442202 or http://dx.doi.org/10.2139/ssrn.3442202 It's no where close to a full study and I lost the impetus/encouragement to do a rigorous experiment, but perhaps it is time that I re-gained some interest in this topic and approach again. Happy to hear from interested and prospective collaborators. Regards Chandra On Sat, Aug 8, 2020 at 11:23 PM Nathaniel Zuk <ZUKN@xxxxxxxx> wrote: > (Today I learned) It's called a mondegreen. > > https://www.newyorker.com/science/maria-konnikova/science-misheard-lyrics-mondegreens > > Nate > ------------------------------ > *From:* AUDITORY - Research in Auditory Perception < > AUDITORY@xxxxxxxx> on behalf of Prof. Roger K. Moore < > 0000011559506d60-dmarc-request@xxxxxxxx> > *Sent:* Friday, August 7, 2020 3:40 PM > *To:* AUDITORY@xxxxxxxx <AUDITORY@xxxxxxxx> > *Subject:* Re: [AUDITORY] Semantic McGurk Effect > > Great example Julia - and, of course, comedians such as Bec Hill ( > https://youtu.be/nEdR44Iftb4) and Peter Kay (https://youtu.be/7my5baoCVv8) > trade on this effect - R > > > > On Fri, 7 Aug 2020 at 14:42, Julia Strand <jstrand@xxxxxxxx> wrote: > > I'm always delighted when auditory phenomena spark the public's interest! > > I wouldn't call this a semantic McGurk, given that it doesn't have to be > driven by simultaneous bottom-up input from two modalities. That is, even > if nothing is written on the screen but you're just thinking "green needle" > to yourself, that's what you're likely to hear (whereas thinking "ga" while > hearing "ba" won't get you to "da" - you need the simultaneous input from > face and voice). So I'd agree with Roger that it's more akin to the phoneme > restoration effect or work like Cynthia Connine's "she ran hot water for > the p/bath," showing how expectations influence interpretation of bottom-up > input. > > I think most of US wouldn't be surprised that the same stimulus can be > perceived in different ways, but my impression is that the general public > tends to believe "what you see is what you get" and underestimates the > power of top-down influences. Same reason #TheDress was such a hit. > > When I include this in my class on speech perception, I also include this video > which shows Grover from Sesame street > <https://languagelog.ldc.upenn.edu/nll/?p=41249> saying EITHER "Yes, yes, > that sounds like an excellent idea!" OR "Yes, yes, that's a f*%#g excellent > idea!" > > Like I'm always telling my students - Speech is hard! Context helps! > > Best, > Julia > > On Fri, Aug 7, 2020 at 4:28 AM Prof. Roger K. Moore < > 0000011559506d60-dmarc-request@xxxxxxxx> wrote: > > I must admit to being surprised by the surprise engendered by this video. > Anyone who was around during the early days of text-to-speech synthesis is > very aware of the danger of presenting the text in advance of or > simultaneous with the generated speech. The intelligibility of the > resulting synthesis could be zero without the 'prior' and 100% with the > visual cue. > > So, given that we know that perception involves the integration of > top-down expectations with bottom-up evidence (going right back to Richard > Warren's work on the 'phoneme restoration effect'), why is this TikTok demo > surprising? Or maybe I'm missing something? > > Best wishes > Roger > > > -------------------------------------------------------------------------------------------- > Prof ROGER K MOORE* BA(Hons) MSc PhD FIOA FISCA MIET > > Chair of Spoken Language Processing > Vocal Interactivity Lab (VILab), Sheffield Robotics > Speech & Hearing Research Group (SPandH) > Department of Computer Science, UNIVERSITY OF SHEFFIELD > Regent Court, 211 Portobello, Sheffield, S1 4DP, UK > > * Winner of the 2016 Antonio Zampolli Prize for "*Outstanding > Contributions * > *to the Advancement of Language Resources & Language Technology * > *Evaluation within Human Language Technologies*" > > e-mail: r.k.moore@xxxxxxxx > web: http://staffwww.dcs.shef.ac.uk/people/R.K.Moore/ > twitter: @xxxxxxxx > Tel: +44 (0) 11422 21807 > Fax: +44 (0) 11422 21810 > Mob: +44 (0) 7910 073631 > > Editor-in-Chief: COMPUTER SPEECH AND LANGUAGE > (http://www.journals.elsevier.com/computer-speech-and-language/) > > -------------------------------------------------------------------------------------------- > > > On Fri, 7 Aug 2020 at 05:12, Malcolm Slaney <malcolm@xxxxxxxx> wrote: > > Has there been anything formal published on this effect? > > https://www.iflscience.com/brain/what-the-hell-is-going-on-in-this-tiktok-audio-illusion > > It sounds to me like a semantic version of the McGurk effect. > > Nice demo. > > - Malcolm > > > > -- > Julia Strand, PhD > Assistant Professor of Psychology > Carleton College > One North College Street > Northfield, MN 55057 > 507-222-5637 > Website <https://apps.carleton.edu/curricular/psyc/jstrand/> > Make an appointment <http://juliastrand.youcanbook.me> > > -- Sharath Chandra Ram PhD Fellow @xxxxxxxx://atec.utdallas.edu/ <https://atec.utdallas.edu/>Machine Listening & Computational Media Art Applied Cognition in HCI & Virtual Environments Information, Media & Technology Policy Alumni@xxxxxxxx School of Informatics Signal Lab : radiospock.tumblr.com Twitter @xxxxxxxx --0000000000004f8d9505ac6aa38a Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr">It's ever so interesting, and I think we had a round o= n a similar illusory conjunction before, with the Laurel-Yanny conundrum.<d= iv><br><div>Incidentally, that same year was the <b>"BrainStorm" = / "Green Needle"</b> conjunction :=C2=A0<a href=3D"https://www.yo= utube.com/watch?v=3D1okD66RmktA">https://www.youtube.com/watch?v=3D1okD66Rm= ktA</a></div><div>In this case you heard what you want to hear! (Don't = bet on it as some folk hear neither=C2=A0of it, or are biased=C2=A0to one o= r the other)</div><div><br></div><div><ul><li>I replicated (with due attrib= ution) and extended upon the Current Biology paper on Laurel_yanny by Press= nitzer & Jackson Graves (et al) if anyone is interested in this non-pee= r reviewed pre-print on SSRN, with interrupted speech perception and correl= ation with hearing loss.=C2=A0</li></ul></div><div><br></div><div><span sty= le=3D"color:rgb(80,80,80);font-family:NexusSansWebPro;font-size:16px">Ramak= rishnan, Sharath Chandra, Illusory Conjunctions in Speech Perception as a L= ens into the Characteristics of Hearing Loss (August 24, 2019). Available a= t SSRN:=C2=A0</span><a href=3D"https://ssrn.com/abstract=3D3442202">https:/= /ssrn.com/abstract=3D3442202</a><span style=3D"color:rgb(80,80,80);font-fam= ily:NexusSansWebPro;font-size:16px">=C2=A0or=C2=A0</span><a href=3D"http://= dx.doi.org/10.2139/ssrn.3442202">http://dx.doi.org/10.2139/ssrn.3442202</a>= <br></div><div><br></div><div>It's no where=C2=A0close to a full study = and I lost the impetus/encouragement to do a rigorous experiment, but perha= ps it is time that I re-gained some interest in this topic and approach aga= in. Happy to hear from interested and prospective collaborators.</div><div>= <br></div><div>Regards</div><div>Chandra</div><div><br></div><div><br></div= ><div><div><br></div><br><div class=3D"gmail_quote"><div dir=3D"ltr" class= =3D"gmail_attr">On Sat, Aug 8, 2020 at 11:23 PM Nathaniel Zuk <<a href= =3D"mailto:ZUKN@xxxxxxxx" target=3D"_blank">ZUKN@xxxxxxxx</a>> wrote:<br></d= iv><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;bord= er-left:1px solid rgb(204,204,204);padding-left:1ex"> <div dir=3D"ltr"> <div style=3D"font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt= ;color:rgb(0,0,0);background-color:rgb(255,255,255)"> (Today I learned) It's called a mondegreen.</div> <div style=3D"font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt= ;color:rgb(0,0,0);background-color:rgb(255,255,255)"> <a href=3D"https://www.newyorker.com/science/maria-konnikova/science-mishea= rd-lyrics-mondegreens" target=3D"_blank">https://www.newyorker.com/science/= maria-konnikova/science-misheard-lyrics-mondegreens</a><br> </div> <div style=3D"font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt= ;color:rgb(0,0,0);background-color:rgb(255,255,255)"> <br> </div> <div style=3D"font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt= ;color:rgb(0,0,0);background-color:rgb(255,255,255)"> Nate</div> <div id=3D"m_-2712768392364833469gmail-m_-6700533499841008357appendonsend">= </div> <hr style=3D"display:inline-block;width:98%"> <div id=3D"m_-2712768392364833469gmail-m_-6700533499841008357divRplyFwdMsg"= dir=3D"ltr"><font face=3D"Calibri, sans-serif" style=3D"font-size:11pt" co= lor=3D"#000000"><b>From:</b> AUDITORY - Research in Auditory Perception <= ;<a href=3D"mailto:AUDITORY@xxxxxxxx" target=3D"_blank">AUDITORY@xxxxxxxx= TS.MCGILL.CA</a>> on behalf of Prof. Roger K. Moore <<a href=3D"mailt= o:0000011559506d60-dmarc-request@xxxxxxxx" target=3D"_blank">0000011= 559506d60-dmarc-request@xxxxxxxx</a>><br> <b>Sent:</b> Friday, August 7, 2020 3:40 PM<br> <b>To:</b> <a href=3D"mailto:AUDITORY@xxxxxxxx" target=3D"_blank">AU= DITORY@xxxxxxxx</a> <<a href=3D"mailto:AUDITORY@xxxxxxxx" = target=3D"_blank">AUDITORY@xxxxxxxx</a>><br> <b>Subject:</b> Re: [AUDITORY] Semantic McGurk Effect</font> <div>=C2=A0</div> </div> <div> <div dir=3D"ltr"> <div dir=3D"ltr"> <div dir=3D"ltr">Great example Julia - and, of course, comedians such as Be= c Hill (<a href=3D"https://youtu.be/nEdR44Iftb4" target=3D"_blank">https://= youtu.be/nEdR44Iftb4</a>) and Peter Kay (<a href=3D"https://youtu.be/7my5ba= oCVv8" target=3D"_blank">https://youtu.be/7my5baoCVv8</a>) trade on this ef= fect - R<br clear=3D"all"> <div> <div dir=3D"ltr"> <div dir=3D"ltr"> <div> <div dir=3D"ltr"> <div> <div dir=3D"ltr"> <div> <div dir=3D"ltr"> <div> <div dir=3D"ltr"> <div> <div dir=3D"ltr"> <div> <div dir=3D"ltr"> <div> <div dir=3D"ltr"> <div> <div dir=3D"ltr"> <div> <div dir=3D"ltr"> <div> <div dir=3D"ltr"> <div> <div dir=3D"ltr"> <div><br> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> <a href=3D"http:///" target=3D"_blank"></a></div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> <br> </div> </div> </div> <br> <div> <div dir=3D"ltr">On Fri, 7 Aug 2020 at 14:42, Julia Strand <<a href=3D"m= ailto:jstrand@xxxxxxxx" target=3D"_blank">jstrand@xxxxxxxx</a>> = wrote:<br> </div> <blockquote style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204= ,204,204);padding-left:1ex"> <div dir=3D"ltr">I'm always delighted when auditory phenomena=C2=A0spar= k the public's interest!=C2=A0 <div><br> </div> <div>I wouldn't call this a semantic McGurk, given that it doesn't = have to be driven by simultaneous bottom-up input from two modalities. That= is, even if nothing is written on the screen but you're just thinking = "green needle" to yourself, that's what you're likely to hear (whereas thinking "ga" while hearing "ba&quo= t; won't get you to "da" - you need the simultaneous input fr= om face and voice). So I'd agree with Roger that it's more akin to = the phoneme restoration effect or work like Cynthia Connine's "she= ran hot water for the p/bath," showing how expectations influence interpretat= ion of bottom-up input. <div><br> </div> <div>I think most of US wouldn't be surprised that the same stimulus ca= n be perceived in different ways, but my impression is that the general pub= lic tends to believe "what you see is what you get" and underesti= mates the power of top-down influences. Same reason #TheDress was such a hit.=C2=A0</div> <div><br> </div> <div>When I include this in my class on speech perception, I also include t= his <a href=3D"https://languagelog.ldc.upenn.edu/nll/?p=3D41249" target=3D"= _blank"> video which shows Grover from Sesame street</a> saying EITHER "Yes, ye= s, that sounds like an excellent idea!" OR "Yes, yes, that's = a f*%#g excellent idea!"</div> <div><br> </div> <div>Like I'm always telling my students - Speech is hard! Context help= s!</div> <div><br> </div> <div>Best,</div> <div>Julia</div> </div> </div> <br> <div> <div dir=3D"ltr">On Fri, Aug 7, 2020 at 4:28 AM Prof. Roger K. Moore <<a= href=3D"mailto:0000011559506d60-dmarc-request@xxxxxxxx" target=3D"_= blank">0000011559506d60-dmarc-request@xxxxxxxx</a>> wrote:<br> </div> <blockquote style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204= ,204,204);padding-left:1ex"> <div dir=3D"ltr">I must admit to being surprised by the surprise engendered= by this video.=C2=A0 Anyone who was around during the early days of text-t= o-speech synthesis=C2=A0is very aware of the danger of presenting the text = in advance of or simultaneous=C2=A0with the generated speech.=C2=A0 The intelligibility of the resulting synthesis=C2=A0could be= zero without the 'prior' and 100% with the visual cue. <div><br> </div> <div>So, given that we know that perception involves the integration of top= -down expectations with bottom-up evidence (going right back to Richard War= ren's work on the 'phoneme restoration effect'), why is this Ti= kTok demo surprising?=C2=A0 Or maybe I'm missing something? <div><br> </div> <div>Best wishes</div> <div>Roger</div> <div><br clear=3D"all"> <div> <div dir=3D"ltr"> <div dir=3D"ltr"> <div> <div dir=3D"ltr"> <div> <div dir=3D"ltr"> <div> <div dir=3D"ltr"> <div> <div dir=3D"ltr"> <div> <div dir=3D"ltr"> <div> <div dir=3D"ltr"> <div> <div dir=3D"ltr"> <div> <div dir=3D"ltr"> <div> <div dir=3D"ltr"> <div> <div dir=3D"ltr"> <div> <div dir=3D"ltr"><font size=3D"1">-----------------------------------------= ---------------------------------------------------<br> Prof ROGER K MOORE* BA(Hons) MSc PhD FIOA FISCA MIET<br> <br> Chair of Spoken Language Processing<br> Vocal Interactivity Lab (VILab), Sheffield Robotics<br> Speech & Hearing Research Group (SPandH)<br> Department of Computer Science, UNIVERSITY OF SHEFFIELD<br> Regent Court, 211 Portobello, Sheffield, S1 4DP, UK</font> <div><font size=3D"1" face=3D"arial, helvetica, sans-serif"><br> </font></div> <div> <div><font face=3D"arial, helvetica, sans-serif" size=3D"1">* Winner of=C2= =A0the 2016 Antonio Zampolli Prize for "<i>Outstanding Contributions= =C2=A0</i></font></div> <div><font face=3D"arial, helvetica, sans-serif" size=3D"1"><i>to the Advan= cement of Language Resources & Language Technology=C2=A0</i></font></di= v> <div><font face=3D"arial, helvetica, sans-serif" size=3D"1"><i>Evaluation w= ithin Human Language Technologies</i>"</font></div> <font size=3D"1"><br> e-mail:=C2=A0 <a href=3D"mailto:r.k.moore@xxxxxxxx" target=3D"_blank= ">r.k.moore@xxxxxxxx</a><br> web:=C2=A0<a href=3D"http://staffwww.dcs.shef.ac.uk/people/R.K.Moore/" targ= et=3D"_blank">http://staffwww.dcs.shef.ac.uk/people/R.K.Moore/</a></font></= div> <div><font size=3D"1">twitter: @xxxxxxxx<br> Tel: +44 (0) 11422 21807<br> Fax: +44 (0) 11422 21810<br> Mob: +44 (0) 7910 073631<br> <br> Editor-in-Chief: COMPUTER SPEECH AND LANGUAGE<br> (<a href=3D"http://www.journals.elsevier.com/computer-speech-and-language/"= target=3D"_blank">http://www.journals.elsevier.com/computer-speech-and-lan= guage/</a>)</font></div> <div><span style=3D"font-size:x-small">------------------------------------= --------------------------------------------------------</span><br> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> <a href=3D"http:///" target=3D"_blank"></a></div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> <br> </div> </div> </div> <br> <div> <div dir=3D"ltr">On Fri, 7 Aug 2020 at 05:12, Malcolm Slaney <<a href=3D= "mailto:malcolm@xxxxxxxx" target=3D"_blank">malcolm@xxxxxxxx</a>> wrote:= <br> </div> <blockquote style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204= ,204,204);padding-left:1ex"> <div>Has there been anything formal published on this effect? <div>=C2=A0 =C2=A0<a href=3D"https://www.iflscience.com/brain/what-the-hell= -is-going-on-in-this-tiktok-audio-illusion" style=3D"color:rgb(17,85,204);f= ont-family:Arial,Helvetica,sans-serif;font-size:small;font-variant-ligature= s:normal;background-color:rgb(255,255,255)" target=3D"_blank">https://www.i= flscience.com/brain/what-the-hell-is-going-on-in-this-tiktok-audio-illusion= </a></div> <div><br> </div> <div>It sounds to me like a semantic version of the McGurk effect.</div> <div><br> </div> <div>Nice demo.</div> <div><br> </div> <div>- Malcolm</div> <div><br> </div> </div> </blockquote> </div> </blockquote> </div> <br clear=3D"all"> <div><br> </div> -- <br> <div dir=3D"ltr"> <div dir=3D"ltr"> <div> <div dir=3D"ltr"> <div> <div dir=3D"ltr"> <div dir=3D"ltr">Julia Strand, PhD <div>Assistant Professor of Psychology</div> <div>Carleton College</div> <div>One North College Street</div> <div>Northfield, MN 55057</div> <div>507-222-5637</div> <div><a href=3D"https://apps.carleton.edu/curricular/psyc/jstrand/" target= =3D"_blank">Website</a></div> <div><a href=3D"http://juliastrand.youcanbook.me" target=3D"_blank">Make an= appointment</a></div> </div> </div> </div> </div> </div> </div> </div> </blockquote> </div> </div> </div> </blockquote></div><br clear=3D"all"><div><br></div>-- <br><div dir=3D"ltr"= ><div dir=3D"ltr"><div><div dir=3D"ltr"><div dir=3D"ltr"><div dir=3D"ltr"><= span style=3D"color:rgb(0,0,0)">Sharath Chandra Ram</span><br></div><div di= r=3D"ltr"><span style=3D"color:rgb(0,0,0)"><br></span></div><div dir=3D"ltr= "><p style=3D"margin-top:0px;margin-bottom:0px;font-family:Calibri,Helvetic= a,sans-serif"><span style=3D"background-color:rgb(255,255,255)"><font color= =3D"#000000"><span style=3D"font-family:Arial,Helvetica,sans-serif"></span>= <span style=3D"font-family:Arial,Helvetica,sans-serif">PhD Fellow @xxxxxxxx</span><= a href=3D"https://atec.utdallas.edu/" style=3D"font-family:Arial,Helvetica,= sans-serif" target=3D"_blank">https://atec.utdallas.edu/</a></font></span><= /p><p style=3D"margin-top:0px;margin-bottom:0px;font-family:Calibri,Helveti= ca,sans-serif"><span style=3D"background-color:rgb(255,255,255)"><font colo= r=3D"#000000"><a href=3D"https://atec.utdallas.edu/" style=3D"font-family:A= rial,Helvetica,sans-serif" target=3D"_blank"></a><span style=3D"font-family= :Arial,Helvetica,sans-serif">Machine Listening & Computational Media Ar= t</span><br style=3D"font-family:Arial,Helvetica,sans-serif"></font></span>= </p><p style=3D"margin-top:0px;margin-bottom:0px;font-family:Calibri,Helvet= ica,sans-serif"><span style=3D"font-family:Arial,Helvetica,sans-serif;backg= round-color:rgb(255,255,255)"><font color=3D"#000000">Applied Cognition in = HCI &=C2=A0Virtual Environments</font></span></p><p style=3D"margin-top= :0px;margin-bottom:0px;font-family:Calibri,Helvetica,sans-serif"><span styl= e=3D"font-family:Arial,Helvetica,sans-serif;background-color:rgb(255,255,25= 5)"><font color=3D"#000000">Information, Media &=C2=A0Technology=C2=A0P= olicy<br></font></span></p><p style=3D"margin-top:0px;margin-bottom:0px;fon= t-family:Calibri,Helvetica,sans-serif"><span style=3D"background-color:rgb(= 255,255,255)"><font color=3D"#000000"><span style=3D"font-family:Arial,Helv= etica,sans-serif">Alumni@xxxxxxxx School of Informatics</span><br style=3D= "font-family:Arial,Helvetica,sans-serif"><span style=3D"font-family:Arial,H= elvetica,sans-serif">Signal Lab :=C2=A0</span><a href=3D"http://radiospock.= tumblr.com/" style=3D"font-family:Arial,Helvetica,sans-serif" target=3D"_bl= ank">radiospock.tumblr.com</a><br style=3D"font-family:Arial,Helvetica,sans= -serif"><span style=3D"font-family:Arial,Helvetica,sans-serif">Twitter @xxxxxxxx= ntSpock</span></font></span></p><div><br></div><div><div><br></div></div><d= iv><br></div><div><span style=3D"color:rgb(68,68,68);font-family:Roboto,Hel= vetica,Arial,sans-serif;font-size:13px"><br></span></div><div><span style= =3D"color:rgb(68,68,68);font-family:Roboto,Helvetica,Arial,sans-serif;font-= size:13px"><br></span></div></div></div></div></div></div></div></div></div= ></div> --0000000000004f8d9505ac6aa38a--