Forming Test Collections with No System Pooling

DMCA

Forming Test Collections with No System Pooling (2004)

Cached

Download Links

Other Repositories/Bibliography

DBLP

by Mark Sanderson , Hideo Joho

Venue:	In Proceedings of SIGIR
Citations:	43 - 1 self

BibTeX

@INPROCEEDINGS{Sanderson04formingtest,
    author = {Mark Sanderson and Hideo Joho},
    title = {Forming Test Collections with No System Pooling},
    booktitle = {In Proceedings of SIGIR},
    year = {2004},
    pages = {33--40}
}

OpenURL

Abstract

Forming test collection relevance judgments from the pooled output of multiple retrieval systems has become the standard process for creating resources such as the TREC, CLEF, and NTCIR test collections. This paper presents a series of experiments examining three different ways of building test collections where no system pooling is used. First, a collection formation technique combining manual feedback and multiple systems is adapted to work with a single retrieval system. Second, an existing method based on pooling the output of multiple manual searches is re-examined: testing a wider range of searchers and retrieval systems than has been examined before. Third, a new approach is explored where the ranked output of a single automatic search on a single retrieval system is assessed for relevance: no pooling whatsoever. Using established techniques for evaluating the quality of relevance judgments, in all three cases, test collections are formed that are as good as TREC.

Keyphrases

test collection single retrieval system collection formation technique different way pooled output system pooling wider range relevance judgment single automatic search ntcir test collection new approach manual feedback standard process multiple manual search multiple system ranked output multiple retrieval system retrieval system test collection relevance judgment

DMCA