Abstract:
Surprising variability emerged when the distribution of the quality ratings that individual voices received in data sets that met conventional standards for rating reliability was examined. Whatever quality was rated or scale used, individual raters were self-consistent in judgments of voices near scale end points, but consistency for nonextreme voices varied widely by voice. Raters differed in how they defined scale endpoints. Pairs of raters did not agree consistently in ratings for any voices, and groups of raters taken together never agreed well in their ratings for any but the most extremely pathologic stimuli. Traditional measures of rating reliability do not document this variability, and may be misleading when used for this purpose. These findings suggest that quality scales are probably better treated as categorical, rather than as interval, with category boundaries defined individually for each listener. Ratings averaged across listeners appear to have little meaning, and ``moderately pathological'' is probably undefinable perceptually. These findings are also consistent with evidence from nonparametric multidimensional scaling [Kreiman and Gerratt, J. Acoust. Soc. Am. (in press)], which suggests that the distinction between severe and mild-to-moderate pathology is the only perceptually valid dimension for pathological voice quality. [Work supported by NIDCD.]