Computers cannot fully understand spoken language without access to the wide range of modalities that accompany speech. This thesis addresses the particularly expressive modality of hand gesture, and focuses on building structured statistical models at the intersection of speech, vision, and meaning. My approach is distinguished in two key respects. First, gestural patterns are leveraged to discover parallel structures in the meaning of the associated speech. This differs from prior work that attempted to interpret individual gestures directly, an approach that was prone to a lack of generality across speakers. Second, I present novel, structured statistical models for multimodal language processing, which enable learning about gesture in its linguistic context, rather than in the abstract. These ideas find successful application in a variety of language processing tasks: resolving ambiguous noun phrases, segmenting speech into topics, and producing keyframe summaries of spoken language. In all three cases, the addition of gestural features – extracted automatically from video – yields significantly improved performance over a state-of-the-art text-only alternative. This marks the first demonstration that hand gesture improves automatic discourse processing.

Abstract. This paper describes a pilot study of gestures which are connected to anaphoric expressions and to their antecedents in video-recordings of spontaneous two- and three party interactions. The record-ings have been transcribed, multimodally annotated and analysed. The results of this analysis and of machine learning experiments run on the annotated data show that gestures which are related to anaphora (or co-referring expression) and to their antecedents have many common shape attributes and values. They also show that shape attributes and values can be used for identifying gestures connected to referring expressions automatically. These results are promising for both anaphora resolution and for the generation of plausible conversational agents.

Abstract: A novel algorithm for head gestures interpretation is designed and tested. The designed system carries out gesture detection and recognition using the MGDA algorithm, which implements random sampling and importance sampling, such technique can track head poses and estimate head positions. Problem statement: Head position is an important indicator of a person’s focus of attention, which can be used as a key for multi-view face analysis assuming that face recognition and identification to be viewed dependently. This will help in selecting the best view model. Also, in the past few years face detection and person identification became important issues due to security concerns, leading to head gesture algorithm development and implementation. Approach: The captured image was allocated a map after which a file conversion process is carried out, allowing the next stage of image data conversion of head poses to be applied. This results in a specific number of matrices per pose that hold the necessary information. Such information was then allocated sequences representing head gesture poses which is combined for classification and correlation purposes to regenerate a predicted time reconstructed continuous movements. Results: A reliable, fast and robust approach for static head gesture recognition was achieved and presented. Conclusion: This very successful approach to head pose detection and gesture classification is strongly supported by its ability to correlate different signal input technologies as the devised algorithm can accommodate different inputs.