We explore several language modeling strategies for increasing the recognition accuracy among large sets of proper nouns in a mapbased multimodal dialogue system which provides restaurant information. In particular, we evaluate several mechanisms for exploiting dialogue context, the two most promising of which involve a semistatic metropolitan-region based large set of proper nouns competing with a smaller, in-focus subset. We show that these techniques decrease word, concept, and proper noun error rates under several training conditions. We also present a technique to generalize sparse training data through derived templates to improve language model robustness. Index Terms — multimodal dialogue system, language modeling, context-sensitive, restaurants, proper nouns 1.

If we can model the cognitive and communicative processes underlying speech, we should be able to better predict what speakers will do, and thus improve language models. This paper presents an initial exploration of this idea. In the Switchboard corpus, we find that word probabilities vary with various non-lexical indicators of cognitive and communicative states, including local volume, local speaking rate and other prosodic features, and also time since start of utterance and since since other reference events. Conditioning word probabilities on 8 such features improved word predictions, reducing the perplexity by 4.4 % relative to a trigram baseline. Key words: dialog state, shallow cognitive state, dialog dynamics, interlocutor behavior,

are those of the author and should not be interpreted as representing the official policies, either express or implied, of any sponsoring institution, the U.S. government, or any other entity. Keywords: spoken dialog systems, conversational spoken language interfaces, error detection, error recovery strategies, error recovery policies, dialog management, RavenClaw, implicitly-supervised One of the most important and persistent problems in the development of conversational spoken language interfaces is their lack of robustness when confronted with understanding-errors. Most of these errors stem from limitations in current speech recognition technology, and, as a result, appear across all domains and interaction types. There are two approaches towards increased robustness: prevent the errors from happening, or recover from them through conversation, by interacting with the users. In this dissertation we have engaged in a research program centered on the second approach. We argue that three capabilities are needed in order to seamlessly and efficiently recover from errors: (1) systems must be able to detect the errors, preferably as soon as they happen, (2) systems must be equipped with a rich repertoire of error recovery strategies that can be used to set the conversation back on track, and (3) systems must know how to choose optimally between different recovery