Abstract

analysis, color segmentation and motion information for finding the positions of whole faces plus the precise location and shape of the mouth. Combining several modalities improves the robustness of the analysis considerably and allows handling a wide variety of conditions. Mouth shapes can enhance the accuracy of speech recognition systems, in particular under noisy conditions. However, finding the shape of the mouth precisely under unrestricted conditions is a challenging task. To be of practical value a system must handle different complexions of people as well as variations in lighting, different head orientations, moustaches, beards, and glasses. To deal with such a diversity of conditions, our system includes several diffkrent models of the face and the mouth area. New faces are compared to these models and the most representative one is chosen for the analysis. We tested our system on samples from video sequences of 50 didrent speakers. When trained on a particular person, the mouth location is found correctly in more than 98% of the images. When trained on a random set of lO people from the database, the system handles typically 87% of the other people correctly. In speakerdependent lip reading experiments we observed 93% word accuracy on five-digit strings.

Citations