This paper reviews a number of recently available techniques in contentanalysis of visual media and their application to the indexing, retrieval,abstracting, relevance assessment, interactive perception, annotation and re-use of visualdocuments. 1. Background A few years ago, the problems of representation and retrieval of visualmedia were confined to specialized image databases (geographical, medical, pilot experimentsin computerized slide libraries), in the professional applications of the audiovisualindustries (production, broadcasting and archives), and in computerized training or education. The presentdevelopment of multimedia technology and information highways has put content processing of visualmedia at the core of key application domains: digital and interactive video, large distributed digital libraries, multimedia publishing. Though the most important investments have been targeted at the information infrastructure (networks, servers, coding and compression, deliverymodels, multimedia systems architecture), a growing number of researchers have realized thatcontent processing will be a key asset in putting together successful applications. The need for contentprocessing techniques has been made evident from a variety of angles, ranging from achievingbetter quality in compression, allowing user choice of programs in video-on-demand, achieving betterproductivity in video production, providing access to large still image databases or integrating still images and video in multimedia publishing and cooperative work. Content-based retrieval of visual media and representation of visualdocuments in human-computer interfaces are based on the availability of content representationdata (time-structure for

A fundamental task in video analysis is to extract structures from the video to facilitate user's access (browsing and retrieval). Motivated by the important role that Table-of-Content (ToC) plays in a book, in this paper we introduce the concept of ToC in the video domain. Some existing approaches implicitly use the ToC, but are mainly limited to low-level entities (e.g. shots and key frames). The drawbacks are that low-level structures (1) contain too many entries to be e ciently presented to the user � and (2) do not capture the underlying semantic structure of the video based on which the user may wishtobrowse/retrieve. To address these limitations, in this paper we present an e ective semantic-level ToC construction technique based on intelligent unsupervised clustering. It has the characteristics of better modeling the time locality and scene structure. Experiments based on real-world movie videos validate the e ectiveness of the proposed approach. Examples are given to demonstrate the usage of the scene based ToC in facilitating user's access to the video. Key words: video accessing, scene level ToC construction 1

. The development of standards for video encoding coupled with the increased power of computing mean that content-based manipulation of digital video information is now feasible. Shots are a basic structural building block of digital video and the boundaries between shots need to be determined automatically to allow for content-based manipulation. A shot can be thought of as continuous images from one camera at a time. In this paper we examine a variety of automatic techniques for shot boundary detection that we have implemented and evaluated on a baseline of 720,000 frames (8 hours) of broadcast television. This extends our previous work on evaluating a single technique based on comparing colour histograms. A description of each of our three methods currently working is given along with how they are evaluated. It is found that although the different methods have about the same order of magnitude in terms of effectiveness, different shot boundaries are detected by the different methods...

The average consumer with a personal computer will soon have access to the world's collections of digital video and images. However, the theory and tools that facilitate browsing, querying, retrieval, and manipulation of imagery are still in their infancy. For example, people would like to access content in movies, e.g. "fast forward to where they bicycle through the sky." This new application area reveals an abundance of unsolved scientific problems for image processing. In this paper I overview key technical challenges that the image processing community should embrace. 1 Introduction For decades, image processing research has focused on problems arising with medical and scientific data, and with compression of "general" imagery such as the ubiquitous Lena images. There has been little change in the emphasis of image processing research. Although tremendous progress has been made, the focus has remained on pixellevel processing, with the goal to acquire, enhance, restore, or compr...

Temporal segmentation of video is a necessary first step to indexing digital video for browsing and retrieval. A number of different video temporal segmentation algorithms have been published in the literature. There has been little effort to evaluate and characterize their performance so as to deliver a single (or set of) algorithms that may be used by other researchers for indexing video databases. We present results of evaluating a number of these algorithms and characterizing their performance, specifically with respect to robustness to encoder and bitrate changes. The lessons learnt have relevance to algorithm development and evaluation in general.

Abstract not found

Extracting information from the ever growing stream of multimedia data is becoming increasingly difficult. One of the main reasons lies within the unstructured way multimedia data are usually presented. Audio-visual material represents a large part of current multimedia material and can be structured in meaningful ways due to the nature of visual communication. This paper surveys several approaches and algorithms that have been recently developed to help in automatically structuring audio-visual data, both for annotation and access

Images or videos may be imaged under different illuminants than models in an image or video proxy database. Changing illumination color in particular may confound recognition algorithms based on color histograms or video segmentation routines based on these. Here we show that a very simple method of discounting illumination changes is adequate for both image retrieval and video segmentation tasks. We develop a feature vector of only 36 values that can also be em used for both these objectives as well as for retrieval of video proxy images from a database. The new image metric is based on a color-channel-normalization step, followed by reduction of dimensionality by going to a chromaticity space. Treating chromaticity histograms as images, we perform an effective low-pass filtering of the histogram by first reducing its resolution via a wavelet-based compression and then by a DCT transformation followed by zonal coding. We show that the color constancy step -- color band normalization -- can...

This paper presents a novel approach to processing encoded video sequences prior to complete decoding. Scene changes are easily detected using DCT coefficients in JPEG and MPEG encoded video sequences. In addition, by analyzing the DCT coefficients, regions of interest may be isolated prior to decompression, increasing efficiency of any subsequent image processing steps, such as edge detection. The results are currently used in a video browser and are part of an ongoing research project in creating large video databases. The procedure is detailed with several examples presented and studied in depth. key words - video processing, scene change detection, video browsing. 1 INTRODUCTION For systems incorporating encoded video, such as video editing systems, various multimedia authoring systems, video-based training systems, and video on demand systems, the ability to manage video efficiently is critical. Although many of these systems incorporate many other types of media as well, manag...

Technological advances over the last several years have resulted in digital image and video libraries that, today, comprise tens of terabytes of on-line data. Due to the continued proliferation of image data, these libraries will grow significantly larger over the next few years. Consequently, the need for databases that can effectively support storage, search, retrieval and transmission of this kind of nontraditional data will grow significantly. This paper describes a project currently in progress under joint sponsorship by NASA, GSFC, that explores some of these challenges. In this paper, we describe the architecture and implementation of a framework to perform content-based search of an image database, where content is specified by the user at one or more of the three abstraction levels: pixel, feature and semantic. This framework incorporates a progressive methodology that allows a computationally efficient implementation of image processing algorithms, yielding the efficient extr...