Abstract:

A method is presented for converting acoustic speech data into a ``speech readable'' movie of a canonical talking face using a neural network. Target values for the network are created by finding the principal components of a set of video frames, with each frame consisting of side-by-side images of the face and tongue of a single speaker. Tongue contours are extracted from mid-sagittal ultrasound images. Input to the network consists of a set of cepstral parameters and their derivatives calculated over 22-ms windows (overlapping by 11 ms). Two such input frames are matched to each video frame (33 ms). Recurrent (backward) connections in the network encourage it to learn not only the acoustic-articulatory pairings, but also articulatory trajectories (expectations of the next articulatory state based on recent articulatory states). It is hypothesized that this supplemental trajectory information helps to alleviate the uncertainties inherent in the vocal tract inverse mapping. After training, the network is presented with new (untrained) tokens (audio only) of utterances from the training corpus and the network's video output is recorded. The network's output is compared to the actual recorded video. [Work supported by US Dept. of Education and NIH.]