Modem large retrieval environments tend to overwhelm their users by their large output. Since all documents are not of equal relevance to their users, highly relevant documents should be identified and ranked first for presentation to the users. In order to develop IR techniques to this direction, it is necessary to develop evaluation approaches and methods that credit IR methods for their ability to retrieve highly relevant documents. This can be done by extending traditional evaluation methods, i.e., recall and precision based on binary relevance assessments, to graded relevance assessments. Alternatively, novel measures based on graded relevance assessments may be developed. This paper proposes three novel measures that compute the cumulative gain the user obtains by examining the retrieval result up to a given ranked position. The first one accumulates the relevance scores of retrieved documents along the ranked result list. The second one is similar but applies a discount factor on the relevance scores in order to devaluate late-retrieved documents. The third one computes the relative-tothe -ideal performance of IR techniques, based on the cumulative gain they are able to yield. The novel measures are defined and discussed and then their use is demonstrated in a case study using TREC data - sample system run results for 20 queries in TREC-7. As relevance base we used novel graded relevance assessments on a four-point scale. The test results indicate that the proposed measures credit IR methods for their ability to retrieve highly relevant documents and allow testing of statistical significance of effectiveness differences. The graphs based on the measures also provide insight into the performance IR techniques and allow interpretation, e.g., from the user point of ...

This paper proposes evaluation methods based on the use of non-dichotomous relevance judgements in IR experiments. It is argued that evaluation methods should credit IR methods for their ability to retrieve highly relevant documents. This is desirable from the user point of view in moderu large IR environments. The proposed methods are (1) a novel application of P-R curves and average precision computations based on separate recall bases for documents of different degrees of relevance, and (2) two novel measures computing the cumulative gain the user obtains by examining the retrieval result up to a given ranked position. We then demonstrate the use of these evaluation methods in a case study on the effectiveness of query types, based on combinations of query structures and expansion, in retrieving documents of various degrees of relevance. The test was run with a best match retrieval system (In- Query ) in a text database consisting of newspaper articles. The results indicate that the tested strong query structures are most effective in retrieving highly relevant documents. The differences between the query types are practically essential and statistically significant. More generally, the novel evaluation methods and the case demonstrate that non-dichotomous rele- vance assessments are applicable in IR experiments, may reveal interesting phenomena, and allow harder testing of IR methods. 1.

The objectives of the study were to conduct a series of observations and experiments under as real-life situation as possible related to: (1) user context of questions in information retrieval; (2) the structure and classification of questions; (3) cognitive traits and decision making of searchers; and (4) different searches of the same question. The study is presented in three parts: Part I presents the background of the study and describes the models, measures, methods, procedures and statistical analyses used. Part II is devoted to results related to users, questions and effectiveness measures, and Part III to results related to searchers, searches and overlap studies. A concluding summary of all results is presented in Part III.

Designing good user interfaces to information retrieval systems is a complex activity. The design space is large and evaluation methodologies that go beyond the classical precision and recall figures are not well established. In this paper we present an evaluation of an intelligent interface that covers also the user-system interaction and measures user's satisfaction. More specifically, we describe an experiment that evaluates: (i) the added value of the semiautomatic query reformulation implemented in a prototype system; (ii) the importance of technical, terminological, and strategic supports and (iii) the best way to provide them. The interpretation of results leads to guidelines for the design of user interfaces to information retrieval systems and to some observations on the evaluation issue.

The selection routine is a formal decision tree that rep-resents the intuitive rules searchers use when they select search keys, textwords or descriptors. The case study method provided the data through: (1) observa-tion of 47 professional online searchers performing their job-related searches; and (2) analysis of verbal and search protocols involved. Each option in the selection of search keys presents the use of a certain combina-tion of textwords and descriptors which searchers choose because of request or database requirements, or because of their own beliefs. The routine delineates the terminological conditions which lead to the selec-tion of each option. It is the first formal presentation of human knowledge that can be incorporated into the knowledge base of intermediary expert systems.

Abstract not found

Effective retrieval of hierarchically structured web documents should exploit the content and structural knowledge associated with the documents. This knowledge can be used to retrieve optimal documents: documents that contain relevant information, and from which users can browse, using the links in these documents, to retrieve further relevant documents. We refer to this approach as focussed retrieval. This paper investigates the effectiveness of a model for the focussed retrieval of hierarchically structured web documents based on the Dempster-Shafer theory of evidence. To allow for focussed retrieval, the representation of a document is defined as the aggregation of the representation of its own content and that of its child documents. To evaluate the model, we constructed a test collection based on a museum web site. From our experiments on this collection, the results show that the Dempster-Shafer theory, in particular, the aggregation, leads to an effective focussed retrieval of hierarchically structured web documents.

A growing body of studies is developing approaches to evaluating human interaction with Web search engines, including the usability and effectiveness of Web search tools. This study explores a user-centered approach to the evaluation of the Web search engine Inquirus – a Web meta-search tool developed by researchers from the NEC Research Institute. The goal of the study reported in this paper was to develop a user-centered approach to the evaluation including: (1) effectiveness: based on the impact of users ' interactions on their information problem and information seeking stage, and (2) usability: including screen layout and system capabilities for users. Twenty-two (22) volunteers searched Inquirus on their own personal information topics. Data analyzed included: (1) user pre- and post-search questionnaires and (2) Inquirus search transaction logs. Key findings include: (1) Inquirus was rated highly by users on various usability measures, (2) all users experienced some level of shift/change in their information problem, information seeking, and personal knowledge due to their Inquirus interaction, (3) different users experienced different levels of change/shift, and (4) the search measure precision did not correlate with other user-based measures. Some users experienced major changes/shifts in various user-based variables, such as information problem or information seeking stage with a search of low precision and vice versa. Implications for the development of user-centered approaches to the evaluation of Web and IR systems and further research are discussed. 2

Abstract not found

... In this paper we show how the method of hierarchical goal decomposition can be used to understand more precisely the knowledge that is required to perform information search tasks. The analysis pinpoints: (1) the critical strategies in the intermediate layers of knowledge that are known by experts searchers; (2) why such knowledge is difficult to acquire by novice searchers; (3) how the analysis provides testable predictions of behavior based on the acquisition of different types of knowledge. We conclude by discussing the advantages provided by hierarchical goal decompositions, and how such an approach can lead to the design of systems and training.