DMCA
Recent advances in the automatic recognition of audiovisual speech
| Venue: | Proceedings of the IEEE |
| Citations: | 172 - 16 self |
BibTeX
@INPROCEEDINGS{Potamianos_recentadvances,
author = {Gerasimos Potamianos and Chalapathy Neti and Guillaume Gravier and Ashutosh Garg and Student Member and Andrew W. Senior},
title = {Recent advances in the automatic recognition of audiovisual speech},
booktitle = {Proceedings of the IEEE},
year = {},
pages = {2003}
}
Share
 |
 |
 |
 |
||
OpenURL
Abstract
Abstract — Visual speech information from the speaker’s mouth region has been successfully shown to improve noise robustness of automatic speech recognizers, thus promising to extend their usability into the human computer interface. In this paper, we review the main components of audio-visual automatic speech recognition and present novel contributions in two main areas: First, the visual front end design, based on a cascade of linear image transforms of an appropriate video region-of-interest, and subsequently, audio-visual speech integration. On the later topic, we discuss new work on feature and decision fusion combination, the modeling of audio-visual speech asynchrony, and incorporating modality reliability estimates to the bimodal recognition process. We also briefly touch upon the issue of audio-visual speaker adaptation. We apply our algorithms to three multi-subject bimodal databases, ranging from small- to large-vocabulary recognition tasks, recorded at both visually controlled and challenging environments. Our experiments demonstrate that the visual modality improves automatic speech recognition over all conditions and data considered, however less so for visually challenging environments and large vocabulary tasks. Index Terms — Audio-visual speech recognition, speechreading, visual feature extraction, audio-visual fusion, hidden Markov model, multi-stream HMM, product HMM, reliability estimation, adaptation, audio-visual databases. I.
Keyphrases
automatic recognition audiovisual speech recent advance hidden markov model audio-visual database visual front end design audio-visual speaker adaptation index term audio-visual speech recognition main area appropriate video region-of-interest automatic speech recognition visual modality human computer interface noise robustness multi-stream hmm visual feature extraction product hmm large vocabulary task modality reliability mouth region linear image transforms present novel contribution audio-visual speech integration automatic speech recognizers reliability estimation audio-visual speech asynchrony audio-visual automatic speech recognition abstract visual speech information bimodal recognition process large-vocabulary recognition task decision fusion combination multi-subject bimodal database audio-visual fusion new work later topic main component
