Download:
|
by Lizhong Wu, Sharon L. Oviatt, Philip R. Cohen
IEEE Transactions on Multimedia
http://www.cse.ogi.edu/CHCC/Personnel/../Publications/MM10075.ps
Add To MetaCart
Abstract:
Abstract---This paper presents a statistical approach to developing multimodal recognition systems and, in particular, to integrating the posterior probabilities of parallel input signals involved in the multimodal system. We first identify the primary factors that influence multimodal recognition performance by evaluating the multimodal recognition probabilities. We then develop two techniques, an estimate approach and a learning approach, which are designed to optimize accurate recognition during the multimodal integration process. We evaluate these methods using Quickset, a speech/gesture multimodal system, and report evaluation results based on an empirical corpus collected with Quickset. From an architectural perspective, the integration technique presented here offers enhanced robustness. It also is premised on more realistic assumptions than previous multimodal systems using semantic fusion. From a methodological standpoint, the evaluation techniques that we describe provide a valuable tool for evaluating multimodal systems. Keywords--- Multimodal integration, Speech recognition,
Citations
|
2961
|
Pattern Classification and Scene Analysis
– Duda, Hart
- 1973
|
|
510
|
On combining classifiers
– Kittler, Hatef, et al.
- 1998
|
|
270
|
The logic of typed feature structures
– Carpenter
- 1992
|
|
179
|
Probabilistic interpretation of feedforward classification network outputs, with relationships to statistical pattern recognition
– Bridle
- 1990
|
|
168
|
QuickSet: Multimodal interaction for distributed
– Cohen, Johnston, et al.
- 1997
|
|
100
|
Integrating and synchronization of input modes during multimodal human-computer interaction
– OVIATT, DEANGELI, et al.
- 1997
|
|
86
|
Mutual disambiguation of recognition errors in a multimodal architecture
– OVIATT
- 1999
|
|
81
|
Unification-based Multimodal Integration
– Johnston, Cohen, et al.
- 1997
|
|
77
|
Multimodal interfaces for dynamic interactive maps
– OVIATT
- 1996
|
|
73
|
Intelligent multi-media interface technology, in Intelli ent User Znte aces
– Neal, Shapiro
|
|
63
|
Integrating simultaneous input from speech, gaze, and hand gestures
– Koons, Sparrell, et al.
- 1993
|
|
54
|
Put that there: Voice and gesture at the graphics interface
– Bolt
- 1980
|
|
53
|
An overview of predictive learning and function approximation
– Friedman
- 1994
|
|
46
|
Large vocabulary continuous speech recognition: A review
– Young
- 1995
|
|
38
|
Interactive simulation in a multi-person virtual world
– Codella, Jalili, et al.
- 1992
|
|
37
|
Toward a multimodal human computer interface
– Sharma, Pavlovic, et al.
- 1998
|
|
36
|
Synergistic USe of direct manipulation and natural language
– COHEN, DALRYMPLE, et al.
- 1989
|
|
28
|
Building an Application Framework for Speech and Pen Input Integration in Multimodal Learning Interfaces
– Vo, Wood
- 1996
|
|
24
|
Adaptive bimodal sensor fusion for automatic speechreading
– Meier, Hurst, et al.
- 1996
|
|
24
|
Multimodal interfaces
– Waibel, Vo, et al.
- 1995
|
|
21
|
Toward natural gesture/speech HCI: A case study of weather narration
– Poddar, Sethi, et al.
- 1998
|
|
19
|
Finger-pointer: Pointing interface by image processing
– Fukumoto, Suenaga, et al.
- 1994
|
|
14
|
Combining neural networks and context-driven search for online, printed handwriting recognition in the Newton
– Yaeger, Webb, et al.
- 1998
|
|
10
|
Integration of eye-gaze, voice and manual response in multimodal user interface
– Wang
- 1995
|
|
8
|
Combining visual and acoustic speech signals with a neural network improves intelligibility
– Sejnowski, Yuhas, et al.
- 1990
|
|
7
|
Data fusion in robotics and machine intelligence
– Abidi, Gonzalez
- 1992
|
|
5
|
From members to teams to committee - a robust approach to gestural and multimodal recognition
– Wu, Oviatt, et al.
- 2002
|
|
1
|
A.Dalke, J.Phillips, M.Zeller, and W.Humphrey, "Speech/gesture interface to a visual computing environment for molecular biologists
– Sharma, Huang, et al.
- 1996
|
