Evaluation plays an important r ole in the advancement of any field. In computer vision, unsupervised segmentation algorithms, although of great interest, often suffer from lack of a well-defined goal and/or explicit ground truth data, thus rendering evaluation difficult. This paper presents a novel method for evaluating such algorithms using a database for which ground truth data is not explicitly available. Unlike methods of evaluation that rely on the existence or creation of explicit ground truth data, the proposed evaluation procedure subjects human observers to a psychovisual test comparing the results of different segmentation algorithms. The test is designed to answer two main questions: does consensus about a `best' segmentation exist, and if it does, what do we learn about segmentation schemes? The results confirm that human subjects are consistent in their judgements, thus allowing meaningful evaluation. The relevance of the procedure for the evaluation of CBIR systems is discussed. 1.

Content-Based Image Retrieval is important for two reasons. First, the oft-cited growth of image archives in many fields, and the rapid expansion of the Web, mean that successful image retrieval systems are fast becoming a necessity if the mass of accumulated data is to be useful. Second, database retrieval provides a framework within which the important questions of machine vision are brought into focus: successful retrieval is likely to require genuine image understanding. In view of these points, the evaluation of retrieval systems becomes a matter of priority. There is already a substantial literature evaluating specific systems, but little high-level discussion of the evaluation methodologies themselves seems to have taken place. In the first part of the report, we propose a framework within which such issues can be addressed, analyse possible evaluation methodologies, indicate where they are appropriate and where they are not, and critique query-by-example and evaluation methodologies related to it. In the second part of the report, we apply the results of this analysis to a particular dataset. The dataset is problematic but typical: no ground truth is available for its semantics. Considering retrieval based on image segmentations, we present a novel method for its evaluation. Unlike methods of evaluation that rely on the existence or creation of ground truth, the proposed evaluation procedure subjects human subjects to a psychovisual test comparing the results of different segmentation schemes. The test is designed to answer two questions: does consensus about a `best' segmentation exist, and if it does, what do we learn about segmentation schemes for retrieval? The results confirm that human subjects are consistent in their judgements, thus allowing meaningful...

Abstract: Content-Based Image Retrieval is important for two reasons. First, the oft-cited growth of image archives in many fields, and the rapid expansion of the Web, mean that successful image retrieval systems are fast becoming a necessity if the mass of accumulated data is to be useful. Second, database retrieval provides a framework within which the important questions of machine vision are brought into focus: successful retrieval is likely to require genuine image understanding. In view of these points, the evaluation of retrieval systems becomes a matter of priority. There is already a substantial literature evaluating specific systems, but little high-level discussion of the evaluation methodologies themselves seems to have taken place. In the first part of the report, we propose a framework within which such issues can be addressed, analyse possible evaluation methodologies, indicate where they are appropriate and where they are not, and critique query-by-example and evaluation methodologies related to it. In the second part of the report, we apply the results of this analysis to a particular dataset. The dataset is problematic but typical: no ground truth is available for its semantics. Considering retrieval based on image segmentations, we present a novel method for its evaluation. Unlike methods of evaluation that rely on the existence or creation of ground truth, the proposed evaluation procedure subjects human subjects to a psychovisual test comparing