K. Sparck Jones and C. van Rijsbergen. Report on the need for and provision of an "ideal" information retrieval test collection. British Library Research and Development Report 5266, Computer Laboratory, University of Cambridge, 1975.  Home/Search   Document Not in Database   Summary   Related Articles   Check

.... a judgment rate of one document per 30 seconds, and judging roundthe clock with no breaks, it would still take more than nine months to judge one topic for a collection of 800,000 documents (the average size of a TREC collection) Instead, modern collections use a technique called pooling [15] to create a subset of the documents (the pool ) to judge for a topic. Each document in the pool for a topic is judged for relevance by the topic author, and documents not in the pool are assumed to be irrelevant to that topic. Before describing the e ects of incompleteness on a test collection, ....

K. Sparck Jones and C. van Rijsbergen. Report on the need for and provision of an \ideal" information retrieval test collection. British Library Research and Development Report 5266, Computer Laboratory, University of Cambridge, 1975.

....every topic. The size of the TREC document sets makes complete judgments utterly infeasible with 800,000 documents, it would take over 6500 hours to judge the entire document set for one topic, assuming each document could be judged in just 30 seconds. Instead, TREC uses a technique called pooling [8] to create a subset of the documents (the pool ) to judge for a topic. Each document in the pool for a topic is judged for relevance by the topic author. Documents that are not in the pool are assumed to be irrelevant to that topic. The judgment pools are created as follows. When participants ....

K. Sparck Jones and C. van Rijsbergen. Report on the need for and provision of an \ideal" information retrieval test collection. British Library Research and Development Report 5266, Computer Laboratory, University of Cambridge, 1975.

....reported are means and two standard errors. This table is taken from [2] dynamic document collections, it becomes intractable to get accurate recall estimates, since they require relevance judgements for the full document collection. To some extend, focused sampling like in the pooling method [9] as used in TREC [19] can reduce assessment cost. The idea is to focus manual assessment on the top documents from several retrieval systems, since those are likely to contain most relevant documents. While some attempts have been made to evaluate retrieval functions without any human judgements ....

K. S. Jones and C. van Rijsbergen. Report on the need for and provision of an "ideal" information retrieval test collection. British Library Research and Development Report 5266, University of Cambridge, Computer Laboratory, 1975.

....manual relevance judgements by experts [1] However, especially for large and dynamic document collections, it becomes intractable to get accurate recall estimates, since they require relevance judgements for the full document collection. To some extend, focused sampling like in the pooling method [9] as used in TREC [19] can reduce assessment cost. The idea is to focus manual assessment on the 1 f used for evaluation # f used for presentation bxx tfc hand tuned bxx 6.26 1.14 46.94 9.80 28.87 7.39 tfc 54.02 10.63 6.18 1.33 13.76 3.33 hand tuned 48.52 6.32 24.61 4.65 6.04 0.92 Table ....

K. S. Jones and C. van Rijsbergen. Report on the need for and provision of an "ideal" information retrieval test collection. British Library Research and Development Report 5266, University of Cambridge, Computer Laboratory, 1975.

....collections and balancing the load across assessors. 3.3 Relevance assessments Relevance judgments are of critical importance to a test collection. For each topic it is necessary to compile a list of relevant documents as comprehensive a list as possible. All TRECs have used the pooling method [9] to assemble the relevance assessments. In this method a pool of possible relevant documents is created by taking a sample of documents selected by the various participating systems. This pool is then shown to the human assessor, who makes a binary (yes no) relevance judgment for each document in ....

K. Sparck Jones and C. van Rijsbergen. Report on the need for and provision of an "ideal" information retrieval test collection. British Library Research and Development Report 5266, Computer Laboratory, University of Cambridge, 1975.

....1611 (21 ) 93 (06 ) ad hoc 7700 1605 92 HP 105 6 .5 3.3 Relevance assessments Relevance judgments are of critical importance to a test collection. For each topic it is necessary to compile a list of relevant documents as comprehensive a list as possible. All TRECs have used the pooling method [8] to assemble the relevance assessments. In this method a pool of possible relevant documents is created by taking a sample of documents selected by the various participating systems. This pool is then shown to the human assessors. The particular sampling method used in TREC is to take the top 100 ....

K. Sparck Jones and C. van Rijsbergen. Report on the need for and provision of an "ideal" information retrieval test collection. British Library Research and Development Report 5266, Computer Laboratory, University of Cambridge, 1975.

....which can only be obtained through subjective, human interaction. Historically, large, realistic collections and queries with proper relevance scores have not been available to researchers. The standard document collections used for IR system evaluation include Cranfield [11] CACM [21] and NPL [52]. All of these collections are less than 10 Mbytes of raw text and are considered tiny compared to current and anticipated IR system requirements. These small research collections simply do not challenge the physical capacities of today s modern computers. CPUs are getting faster, main memories ....

K. Spark Jones and C. A. Webster. Research in relevance weighting. British Library Research and Development Report 5553, Computer Laboratory, University of Cambridge, 1979.

....using different topic sections is given in section 5. 3.3 Relevance assessments Relevance judgments are of critical importance to a test collection. For each topic it is necessary to compile a list of relevant documents as comprehensive a list as possible. All TRECs have used the pooling method [7] to assemble the relevance assessments. In this method a pool of possible relevant documents is created by taking a sample of documents selected by the various participating systems. This pool is then shown to the human assessors. The particular sampling method used in TREC is to take the top 100 ....

K. Sparck Jones and C. van Rijsbergen. Report on the need for and provision of an "ideal" information retrieval test collection. British Library Research and Development Report 5266, Computer Laboratory, University of Cambridge, 1975.

....about each topic, in particular, to include with each topic a clear statement of what criteria make a document relevant. For each topic it was necessary to compile a list (as comprehensive as possible) of relevant documents. All the TRECs have used the pooling method to compile this list [Jones 75] In this method a pool of possible relevant documents is created by taking a sample of documents selected by the various participating systems. This sample is then shown to the human assessors. The particular sampling method used in TREC is to take the top X documents retrieved by each system ....

Jones, Karen S., and Van C. Rijsbergen. Report on the Need for and Provision of an "Ideal" Information Retrieval Test Collection, British Library Research and Development Report 5266, Computer Laboratory, University of Cambridge, 1975.

No context found.

K. Sparck Jones and C. van Rijsbergen. Report on the need for and provision of an "ideal" information retrieval test collection. British Library Research and Development Report 5266, Computer Laboratory, University of Cambridge, 1975.

No context found.

K. Sparck Jones and C. van Rijsbergen. Report on the need for and provision of an \ideal" information retrieval test collection. British Library Research and Development Report 5266, Computer Laboratory, University of Cambridge, 1975.

No context found.

K. Sparck Jones and C. van Rijsbergen. Report on the need for and provision of an "ideal" information retrieval test collection. British Library Research and Development Report 5266, Computer Laboratory, University of Cambridge, 1975.

No context found.

K. Sparck Jones and C. J. van Rijsbergen. Report on the need for and provision of an "ideal" information retrieval test collection. Technical report, British Library Research and Development Report 5266, Computer Laboratory, University of Cambridge, 1975.

No context found.

K. Sparck Jones and C. van Rijsbergen. Report on the need for and provision of an "ideal" information retrieval test collection. British Library Research and Development Report 5266, Computer Laboratory, University of Cambridge, 1975.

No context found.

K. S. Jones and C. van Rijsbergen. Report on the need for and provision of an "ideal" information retrieval test collection. British Library Research and Development Report 5266, University of Cambridge, Computer Laboratory, 1975.

No context found.

K. Sparck Jones and C. van Rijsbergen. Report on the need for and provision of an \ideal" information retrieval test collection. British Library Research and Development Report 5266, Computer Laboratory, University of Cambridge, 1975.

Online articles have much greater impact   More about CiteSeer.IST   Add search form to your site   Submit documents   Feedback  

CiteSeer.IST - Copyright Penn State and NEC