Subject: Re: AUDITORY Digest - 28 Sep 2006 to 29 Sep 2006 (#2006-215) From: Jont Allen <jontalle@xxxxxxxx> Date: Sat, 30 Sep 2006 06:35:18 -0500Dear Laszlo, Christine is of course correct. I would like to post 3 of many refs. For more, look in my book "Articulation and Intelligibility" published by Morgan Claypool, 2005. If you are finding differently for an ASR system, then that just shows that the HMM "Gain" is turned up way too high. By that I mean, its ignoring the input to some extent, and looking for words that it can put together that make some sense, and thus have a combined low entropy (independent of other phones that it recognizes). Let me give an example: What if the spoken utterance was: "The make had a blue type" and assume the recognizer got, at the phone level: "the make had a blue type" (100% correct), then the recognizer would report: "the man had a blue tie" Get it? What if your relative calls you on the phone, and leaves a message, that gets transcribed "a large ant was fried" and then you listen to the message, and it is really "your great aunt just died" That wouldnt be too good, would it. Maybe not a very good example. They are better taken off of a real system. All the example I have are my personal phone messages (text and wav files), and they have things in them I cant make public. But they can be pretty funny sets of errors, I'll tell you! Tell us what really happened, please. I dont care how off topic it is. Its not off topic, IMO. A comment: It is my opinion that ASR people will not report the phone scores because they dont want their funding sources to dry up. Typically these phone scores are quite low (compared to human scores, that is), being in the 50-75% range, with no noise. When the SNR gets "down" to +10, things are falling appart, and at 0 dB SNR, the scores (in one case I know) are below chance. Yes below chance! Human phone error rates start at somewhere between 1.5-2 % error in quiet. At +10 dB SNR (AI~0.5), the Miller Nicely phone error rate was about 10%. At 0 dB the AI is about 0.2 (Allen 2005, JASA, Fig. 6) which gives a phone error rate of about 30%. The 50% point is about -6dB SNR, and an AI of about 0.06. We have unpublished results (in review) where we repeated some of this and found 2% error in quiet (consonants scored from CVs), 10% at -6 dB SNR, and 50% error at -18 dB SNR. However, we found that there are 3 sets of consonants, with one group of 5 consonants, having very high error. These bias the average numbers way up. The rest of the sounds (11 of them) are much better than what I quote above. One group has an error of 0.5% error in quiet (5 errors per 1000 presentations). I have run on too long. Please tell us more! Jont Allen REFS: @xxxxxxxx{Bronkhorst93, author={Bronkhorst, A. W. and Bosman, A. J. and Smoorenburg, G. F.}, title={A model for context effects in speech recognition}, journal=JASA, year={1993}, month=jan, volume={93}, number={1}, pages={499-509}, note_={} } @xxxxxxxx{Boothroyd88, author={Boothroyd, A. and Nittrouer, S.}, year={1988}, title={Mathematical treatment of context effects in phoneme and word recognition}, journal={J. Acoust. Soc. Am.}, volume={84}, number={1}, pages={101-114} } @xxxxxxxx{Boothroyd93, author={Boothroyd, A.}, title={Speech preception, sensorineural hearing loss, and hearing aids}, booktitle={Acoustical Factors affecting Hearing aid performance}, editor={Studebaker, G. A. and Hochberg, I.}, publisher={Allyn and Bacon}, address={Boston}, year={1993}, pages={277-299}, note_={} } AUDITORY automatic digest system wrote: > > Date: Fri, 29 Sep 2006 12:37:34 +0200 > From: Toth Laszlo <tothl@xxxxxxxx> > Subject: reference needed (ASR) > > Dear List, > > I know that speech recognition is a bit off-topic here, but I don't know > of a more proper place to ask this. A reviewer wrote to a paper of > mine that "the fact that better phone recognition does not necessarily > mean better word recognition is already known, and people have been > talking about it very frequently. This should be made clear and perperly > referenced in the paper". Unfortunately, I'm personally sure that I've > never seen this written down, because it would have saved me a lot of > work -- but, unfortunately, I had to learned it from my own failures, > so I'm sure I won't be able to recall any references for this. I'm also > unable to figure out how to turn this thing into a reasonable Google > search term (actually, I've just managed to find a reference for just the > opposite - that "better phone recognition undoubtedly leads to better word > recognition"). So, if anyone can tell me any paper stating or showing > results that "better phone recognition does not necessarily mean better > word recognition", I would be very grateful. > Thanks, > > Laszlo Toth > Hungarian Academy of Sciences * > Research Group on Artificial Intelligence * "Failure only begins > e-mail: tothl@xxxxxxxx * when you stop trying" > http://www.inf.u-szeged.hu/~tothl * > > ------------------------------ > > Date: Fri, 29 Sep 2006 07:16:41 -0400 > From: Christine Rankovic <rankovic@xxxxxxxx> > Subject: Re: reference needed (ASR) > > The statement of the reviewer--that better phone recognition does not mean > better word recognition--is wrong. It is possible that the reviewer could > support this statement with data from poorly conducted speech recognition > tests like, for example, those conducted with an inadequate number of speech > items, or when mean scores comprise scores of too few listeners. > > Christine Rankovic > >