How reliable are the results of large-scale information retrieval experiments (1998) [64 citations — 2 self]
Popular Tags
No tags have been applied to this document.
BibTeX | Add To MetaCart
@INPROCEEDINGS{Zobel98howreliable,author = {Justin Zobel},
title = {How reliable are the results of large-scale information retrieval experiments},
booktitle = {Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval},
year = {1998},
pages = {307--314},
publisher = {ACM Press}
}
Years of Citing Articles
Abstract:
Two stages in measurement of techniques for information retrieval are gathering of documents for relevance assessment and use of the assessments to numerically evaluate effectiveness. We consider both of these stages in the context of the TREC experiments, to determine whether they lead to measurements that are trustworthy and fair. Our detailed empirical investigation of the TREC results shows that the measured relative performance of systems appears to be reliable, but that recall is overestimated: it is likely that many relevant documents have not been found. We propose a new pooling strategy that can significantly increase the number of relevant documents found for given effort, without compromising fairness. 1
