In designing Jabberwocky---a speech-based interface to Microsoft PowerPoint---we have tried to go beyond simple commands like "Next slide, please" and make a tool that aids speakers as they present and even learns as they rehearse their presentations. Jabberwocky looks at the contents of the slides, extracting key words and phrases and associating them with their places in the presentation. By listening for these phrases (and synonymous phrases derived using syntactic rules) Jabberwocky is able to follow along with the presentation, switching slides at the appropriate moments. In this paper, we discuss the implementation of this system---a component of our Intelligent Classroom project---and look at how we are using it.

Intelligent environments are an interesting development and research application problem for multiagent systems. The functional and spatial distribution of tasks naturally lends itself to a multi-agent model and the existence of shared resources creates interactions over which the agents must coordinate. In the UMASS Intelligent Home project we have designed and implemented a set of distributed autonomous home control agents and deployed them in a simulated home environment. Our focus is primarily on resource coordination, though this project has multiple goals and areas of exploration ranging from the intellectual evaluation of the application as a general MAS testbed to the practical evaluation of our agent building and simulation tools. 1 Introduction The intelligent home project (IHome) at the UMASS multi-agent systems lab is an exploration in the application of multi-agent systems technology to the problem of managing an intelligent environment. We have implemented a sophisticate...

In this paper we discuss our use of multi-modal input to improve human computer interaction. Specifically we look at the methods used in the Intelligent Classroom to combine multiple input modes, and examine in particular the visual input modes. The Classroom provides context that improves the functioning of the visual input modes. It also determines which visual input modes are needed when. We examine a number of visual input modes to see how they fit into the general scheme, and look at how the Classroom controls their operation.

The ability to reason about the activity of a user is crucial to the implementation of any Intelligent User Interface. If it is able to recognize what a user is doing, a computer can act to cooperate. Most computer systems limit themselves to command-response interactions---their trivial understandings of their users cannot support a more complicated interaction. However, by looking at the tasks that their users are performing and reasoning about sequences of actions, a computer system can provide a more interesting level of interaction that is more efficient and does not demand as much of its users. Furthermore, the understanding of the user's activity provides a context within which to better understand future actions and to tune the sensing systems to look and listen for the actions that the user is most likely to take next. Finally, in many domains, such computer systems can recognize user tasks and act to cooperate without requiring a deep, goal-oriented understanding. In this paper, we look at the process-based interface used in the Intelligent Classroom, focusing on how a human lecturer can control it by simply going about her presentation. Also, we look at how the general ideas have been adapted to Jabberwocky, a speech-based interface to Microsoft PowerPoint that automatically switches slides, and how they are being applied to extend the functionality of Watson, an autonomous web research tool that uses the document a user is viewing as a search context.

In the software industry, designers are forever trying to "improve " their products by adding ever more features to them, producing bloated software systems that are capable of doing just about anything. However, these systems often make it increasingly difficult for their users to perform their tasks as they are forced to wade through a mess of unwanted features to find the few that they actually need. We believe that a fruitful area of research is in building intelligent systems for particular tasks and then having the systems actively try to assist their users in performing these tasks. Such a system knows the plans (and the problems associated with those plans) that their users are likely to pursue. These competent assistants can use their expertise on their particular tasks to guide their users through their tasks, provide better help, or perhaps even volunteer to take over some part of the task. Such an assistive agent is able to helpful because, through its task knowledge, it is able to limit what it needs to consider in cooperating with its user. In this paper we look at the implementation of a competent assistant that functions in a physical domain. The Intelligent Classroom is a prototype automated lecture facility that serves as its own audio/visual assistant. We focus on the representations and algorithms to use task knowledge to produce cooperative behavior, arguing these techniques could easily to be extended for use in a wide range of domains (i.e. both physical and purely electronic domains).

Progress in computer vision and speech recognition technologies has recently enabled multimodal interfaces that use speech and gestures. These technologies o#er promising alternatives to existing interfaces because they emulate the natural way in which humans communicate. However, no systematic work has been reported that formally evaluates the new speech/gesture interfaces. This paper is concerned with formal experimental evaluation of new human-computer interactions enabled by speech and hand gestures.

The Intelligent Classroom is an automated lecture facility where one of the primary goals is that speakers be able to control it by interacting with it as they would with a human A/V technician. In this paper we describe our research in imbedding Microsoft Powerpoint into the Intelligent Classroom. In particular we discuss howwe use two modes of sensing (Computer Vision and Speech Recognition) to provide hands-free control of slide presentations. We look at utterances and gestures that serve as commands. And, more interestingly, we discuss some probabilistic techniques that can be used to match slide content to the speaker's words, providing new and useful ways of performing a presentation.

This paper is an adaptation of an article that appeared in the September/October 1999 issue of the IEEE Intelligent Systems journal. It provides an informal description of the Intelligent Classroom and looks at examples of what happens as the speaker writes on the board, lectures from slides, and does an anatomy lecture. Also, the paper features pretty color pictures.

Ambient intelligence is an emerging discipline that brings intelligence to our everyday environments and makes those environments sensitive to us. Ambient intelligence (AmI) research builds upon advances in sensors and sensor networks, pervasive computing, and artificial intelligence. Because these contributing fields have experienced tremendous growth in the last few years, AmI research has strengthened and expanded. Because AmI research is maturing, the resulting technologies promise to revolutionarize daily human life by making people’s surroundings flexible and adaptive. In this paper we provide a survey of the technologies that comprise ambient intelligence and of the applications that are dramatically affected by it. In particular, we specifically focus on the research that makes AmI technologies “intelligent”. We also highlight challenges and opportunities that AmI researchers will face in the coming years.

We describe an automatic classroom capture system that detects and records significant (stable) points in lectures by sampling and analyzing a sequence of screen capture frames from a PC used for presentations, application demonstrations, etc. The system uses visual inspection techniques to scan the screen capture stream to identify points to store. Unlike systems that only detect and store slide presentation transitions, this system detects and stores significant frames in any style of lecture using any program. The system is transparent to the lecturer and requires no software or training. It has been tested extensively on lectures with multiple applications and pen-based annotations and has successfully identified “significant” frames (frames that represent stable events such as a new slide, bullet, figure, inked comment, drawing, code entry, application entry etc.). The system can analyze over 20000 frames and typically identifies and stores about 100 significant frames within minutes of the end of a lecture. A time stamp for each saved frame is recorded and will in the future be used to compile these frames into a jMANIC [16] multimedia record of the class.