W. Wahlster, E. Andre, W. Graf, and T. Rist, "Designing Illustrated Texts," Proceedings of EACL'91, Berlin, Germany, 1991.  Home/Search   Document Not in Database   Summary   Related Articles   Check

.... the shape of an object, on the other hand, they tend to redundantly use their hands and their speech to describe the thing [5] In this sense, the semantic and pragmatic compatibility seen in the gesture speech relationship recalls the interaction of words and graphics in multimodal presentations [8, 10, 27]. In fact, some suggest [16] that gesture and speech arise together from an underlying representation that has both visual and linguistic aspects, and so the relationship between gesture and speech is essential to the production of meaning and to its comprehension. As far as choice of modalities ....

W. Wahlster, E. Andre, W. Graf, and T. Rist, "Designing Illustrated Texts," Proceedings of EACL'91, Berlin, Germany, 1991.

....equipment, answering questions or giving tutorials. To date, the conversational behaviors of their agents are limited to greeting and farewell rituals, gaze, pointing gestures and body positioning. In another vein entirely, research on generating words and graphics in multimodal presentations [10 12] [ Kerpedjiev and Roth, this volume] examines the different affordances of language and pictures, and reminds us that in writing as in speaking communicative goals may be differentially mapped onto different modalities. 2.1 User Studies on Embodied Interface Agents Koda and Maes [13] and ....

Wahlster, W., et al. Designing illustrated texts. in EACL'91. 1991.

....or the respective locations of two objects may be expressed by the position of the two hands. The semantic and pragmatic compatibility seen in the gesture speech relationship recalls the interaction of words and graphics in multimodal presentations (Feiner and McKeown 1991; Green et al. 1998; Wahlster et al. 1991). In storytelling, underlying narrative structure may be indexed by differential use of gesture: iconic gestures tend to occur with plot advancing description of the action, deictic gestures with the introductionof new characters, and beat gestures at the boundaries of episodes (Cassell and ....

....the different modalities on a par with one another. Such representations have been explored in research on combining linguistic and graphical interaction. For example, multimodal managers have been described to allocate an underlying content representation for generation of text and graphics (Wahlster et al. 1991; Green et al. 1998) Meanwhile, Johnston et al. 1997; Johnston 1998) describe a formalism for tightly coupled interpretation which uses a grammar and semantic constraints to analyze input from speech and pen. But spontaneous gesture requires a distinct analysis. For example, we need some notion ....

W. Wahlster, E. Andre, W. Graf, and T. Rist. Designing illustrated texts. In Proceedings of EACL, pages 8--14, 1991.

....was that we lacked two functions in our system: first, a multimodal manager that distributes meaning across the modalities, but that is essentially modality independent in its functioning. Such managers have been described for multimodal integration for generation of text and graphics [32], and multimodal integration in input [12] Second, we lacked an understanding of what shape a particular gesture would take: how did we describe which particular gesture would be generated This is similar to the problem of word choice in text generation (Elhadad et al. to appear) We will ....

Wahlster, W., E. Andr, W. Graf, & T. Rist, "Designing Illustrated Texts," Proceedings of 5th EACL, Berlin, Germany, 1991.

....speech, then the system will search for the missing information in the gestures and or gaze. Time stamps unite the actions in the different modalities into a coherent picture. Wahlster used a similar method, also depending on the linguistic input to guide the interpretation of the other modalities [27]. Bolt and Herranz described a system that allows a user to manipulate graphics with two handed semiiconic gesture [6] Using a cutoff point and time stamping, motions can be selected that relate to the intended movement mentioned in speech. Sparrell used a scheme based on stop motion analysis: ....

Wahlster, W., André, E., Graf, W. and Rist, T. Designing illustrated texts. In Proceedings of the 5th EACL (Berlin, Germany, April 1991), 8-14

....speech, the system would search for the missing information in the gestures and or gaze. Time stamps united the actions in the different modalities into a coherent picture. Wahlster used a similar method, also depending on the linguistic input to guide the interpretation of the other modalities (Wahlster et al. 1991). Bolt and Herranz describe a system that allowed a user to manipulate graphics with two handed semi iconic gesture (Bolt and Herranz 1992) Using a cutoff point and time stamping, motions could be selected that related to the intended movement mentioned in speech. Sparrell used a scheme based on ....

Wahlster, W., André, E., Graf, W. and Rist, T. Designing illustrated texts. In Proceedings of the 5th EACL (Berlin, Germany, April 1991), 1991, 8-14.

....may be conveyed by gesture. Thus simultaneity of two events, or the respective locations of two objects may be expressed by the position of the two hands. In this sense, the gesture speech relationship resembles the interaction of words and graphics in the generation of multimodal text ( 6] [21]) In storytelling, narrative structure may be indexed by differential use of gesture: iconic gestures tend to occur with plotadvancing description of the action, deictic gestures with the introduction of new characters, and beat gestures at the boundaries of episodes ( 3] We propose to use the ....

W. Wahlster, E. Andr e, W. Graf, and T. Rist. Designing illustrated texts. In Proceedings of the 5th EACL, pages 8--14, 1991.

....As in Put that There, speech drives the analysis of the gesture: if information ismissing from speech (e.g. delete that one ) then the system will search for the missing information in the gestures and or gaze. Time stamps unite the actions in the different modalities into a coherent picture. (Wahlster, 1991) uses a similar method, also depending on the linguistic input to guide the interpretation of the other modalities. Bolt Herranz, 1992) describe a system that allows a user to manipulate graphics with semi iconic gestures. Bers, in press) developed a system that allows the user to combine ....

Wahlster, W., Andre, E., Graf, W. & Rist, T. (1991). Designing illustrated texts.Proceedings of the Fifth EACL: 8-14.

Online articles have much greater impact   More about CiteSeer.IST   Add search form to your site   Submit documents   Feedback  

CiteSeer.IST - Copyright Penn State and NEC