Voorhees, E. M. & Harman, D. (1997). Overview of the Fifth Text REtrieval conference (TREC-5). In Harman, D. (Ed.), TREC-5, Proceedings of the Fourth Text Retrieval Conference (pp.1-28). Washington, DC: Government Printing Office.  Home/Search   Document Not in Database   Summary   Related Articles   Check

....following section we present a method for creating simulated test collections of structured documents that allow such an investigation. 3 Automatic Construction of Structured Document Test Collections Although many test collections are composed of documents that contain some internal structure [19, 1], relevance judgements are usually made at the document level (root contexts) or at the atomic context level. This means that they cannot be used for the evaluation of structured document retrieval systems, which would require relevance judgements at the root, atomic and inner levels. Our ....

VOORHEES, E., AND HARMAN, D. Overview of the Fifth Text REtrieval Conference (TREC-5). In Proceedings of the 5th Text Retrieval Conference (Gaitherburg, 1996), pp. 1-29.

....evidence, section 3.3.2, and finally results of combining evidence from the term characteristics, section 3.3.3. 3.3. 1 Experimental setup In these experiments we used the Wall Street Journal (1990 92) WSJ) and the Associated Press (1988) AP) test collections from the TREC 5 set of collections (Voorhees and Harman, 1996). The details of these collections are summarised in Table 9. We applied common IR indexing steps such as the removal of highly frequent terms and the reduction of terms to their root variant (Van Rijsbergen, 1979) Collection AP WSJ Number of documents 79 919 74 580 Number of queries used ....

E. M. Voorhees and D. Harman. Overview of the Fifth Text REtrieval Conference

.... is a significant increase in retrieval effectiveness (for more details, see Sections 4 and 5) Pecent experiments by several groups participating in TPEC also suggest that, averaged over large query sets (with 50 queries) adhoc expansion yields significant improvements in overall performance [20, 1, 4, 11]. Thus, it seems worthwhile to continue using adhoc expansion in retrieval for short queries, while exploring ways to prevent quer l drift the alteration of the focus of a search topic caused by improper expansion (as described in the above example) Since the presence of a large proportion of ....

....investigated more extensively in the next section. 4 Experiments and Results In order to determine the usefulness of the techniques described above, we test them on a variety of tasks. We use the TREC collections in our experiments. Our methods are evaluated on the adhoc tasks for TRECs 3 6 [8, 9, 20, 21]. The query sets and document collections used in these tasks are shown in Table 1. Since we are interested in studying short queries, we use only the Description field for queries 151 300. For the TREC 6 queries (numbered 301 350) we use the Title field in addition a. Our experiments use ....

E. M. Voorhees and D. K. Harman. Overview of the Fifth Text REtrieval Conference (TREC-5) . In E. M. Voorhees and D. K. Harman, editors, Proceedings of the Fifth Text REtrieval Conference (TREC-5). NIST Special Publication 500-238, 1997.

....eliminate caching e#ects. The system used was the prototype text retrieval engine MG [3, 35] with modifications to allow DO processing. We are confident (after analysis of several false starts) that the implementation in each case is of good quality. The database used was disks 2 and 4 of TREC [31], which together contain about 530,000 documents. The queries used were the full text of topics 251 300; after stemming, casefolding, and elimination of duplicate terms, their average length is approximately 30 terms. To simulate the e#ect of increasing query length, we generated 1 word queries by ....

E. Voorhees and D. Harman. Overview of the Fifth Text REtrieval Conference (TREC-5). In D.K. Harman, editor, Proceedings of the Fifth Text REtrieval Conference (TREC-5), pages 1--28, 1996.

....ranking based on the pivoted cosine measure is expected to perform reasonably well [38] We applied the same set of experiments to a larger text collection with more uniform document lengths. Two full disks of TREC data were selected (TREC 24) the test data used for the TREC 5 conference [43]. The query set contained 50 topics, numbered from 251 to 300. The pivoted cosine measure was used for whole document, paragraphs, pages,andtiles ranking. windows were ranked with the cosine measure without length normalisation. The experimental results for the TREC 24 collection are summarised ....

E. M. Voorhees and D. Harman. Overview of the Fifth Text REtrieval Conference (TREC-5). In D. K. Harman, editor, Proceedings of the 5th Text REtrieval Conference (TREC-5),NIST Special Publication 500-238, pages 1--28, Nov. 1996.

....to each query. The system resolves each query and is scored according to its ability to fetch the relevant documents. It is well known that the reliability of measurement of a system depends on the quality of the relevance judgements, and that relevance assessors are rarely in exact agreement [4, 5, 7, 11, 14, 15]. Such human factors problems can introduce error into information retrieval experiments, but, assuming the assessment is su#ciently careful (that the assessor, for example, has not simply checked whether the query terms occur in each document) they should not in the general case introduce bias ....

....10 20 30 40 New relevant documents Figure 2: Total number of new relevant documents at each pool depth, actual and estimated, for queries 251 300 from TREC 5. On left, depths 3 100. On right, depths 80 100, expanded to show detail. last system identifies over 200 new relevant documents [14]. To address this di#culty we generated a series of over 1000 random permutations of the list of systems and averaged the number of new documents introduced by the kth system over all the permutations. This average behaviour is plotted for TREC 5 in Figure 3, together with a curve fitted as in ....

E. Voorhees and D. Harman. Overview of the fifth text retrieval conference (TREC-5). In E. Voorhees and D. Harman, editors, Proc. Text Retrieval Conference (TREC), November 1996.

....of schemes S 1 and S 2 , for a topic wise comparison, we define the effectiveness of data fusion E (S 1 f S 2 ) as: 2. 2 Data Sets We use the output lists of the IR schemes produced for the routing tasks of the fourth and fifth Text REtrieval Conferences (i.e. TREC 4 and TREC 5, see Harman 1996; Voorhees Harman 1997) as data for training and testing respectively. In the TREC 4 routing task, there were 26 schemes run on the full document collection for 50 topics, producing 16,250 cases of pairwise data fusion. In TREC 5 routing task, there were 23 schemes for 50 topics, however, 5 of the topics (topics 68, ....

Voorhees, E.M. & Harman, D. (1997). Overview of the Fifth Text REtrieval Conference. In D.

....stored pairs of doc ID and weight (in the general case) appearing in term lists. This means that I was over 1GB in size and was not memory resident, especially for the lowpowered machines considered typical by [MZ96] To give a second example, in [KZS99] about 530,000 documents from TREC 5 [VH96] were used in testing, and these documents were broken into smaller documents of 50 500 bytes each t o give a collection of 7.7 million small documents. From these two examples, we see that CPU cache hits will not be an important performance consideration in accesses t o Accumulators in Algorithm ....

E. Voorhees and D. Harmon. Overview of the Fifth Text REtrieval Conference (TREC-5). Proceedings of the 5th Text Retrieval Conference, Nov. 1996, NIST, http://trec.nist.gov/pubs.html

....either by top level categories or by second level categories; i.e. either SP # fC 0 ;C 1 ; C 9 g,orSP # fC 00 ;C 01 ; C 99 g. 4.3. Methodology and performance metrics We chose to base our evaluation methodology on the one used in the routing track of the TREC benchmark suite [24]. The idea is to have the system score and then rank order a collection of documents based on their likelihood of relevance to a particular profile. The experiments are executed as follows. Each run starts by randomly selecting categories to form a synthetic user profile of desired complexity. The ....

....management component. To date, however, these projects have not emphasized learning based acquisition and maintenance of profiles. There has been significant research on text based profile construction in information retrieval community (e.g. 5, 1, 7, 3] especially in the framework of TREC [24]. The main emphasis of TREC, however, has always been on the effectiveness of the participating systems, rather than on their efficiency. Most of the techniques used for these tasks require batch processing of previously judged documents, imposing relatively high storage and computation costs, and ....

E. M. Voorhees and D. Harman. Overview of the fifth Text REtrieval Conference (TREC-5). In The Fifth Text REtrieval Conf.. (TREC-5), NIST, Gaithersburg, 1996.

....years, the annual TREC (Text REtrieval Conference) large scale blind experiments sponsored by NIST and DARPA have given immense impetus to IR research. During TREC 5 6, monolingual Chinese retrieval was investigated with a fairly large GB encoded Chinese collection of 170 MB in size and 54 queries (Voorhees Harman 1997, 1998) Accompanying each Chinese query is an English counterpart. Although these are not exact translations, they carry the same meaning pretty closely, and we will regard them as a standard English query set to start with. Each query also has a set of Chinese answer documents that have been ....

Voorhees, E.M. & Harman, D.K (1997). Overview of the Fifth Text REtrieval Conference (TREC-5). In: Information Technology: The Fifth Text REtrieval Conference (TREC-5), E.M.Voorhees & D.K. Harman, (eds.), NIST SP 500-238, pp.1-28. GPO: Washington, D.C.

....Jones van Rijsbergen, 1975) to select a sample of the participants results and provides these documents to the users for relevance judgments. The users judgments serve as the basis for relevance on these topics, both for evaluation in a given TREC and as part of the permanent test collection (Voorhees Harman, 1997). The assessors used the ZPRISE interface during the topic development task and the relevance assessment task. The ZPRISE system, originally known as PRISE, developed at NIST in 1988 for the IRS as a prototype experimental statistical full text searching system, demonstrated the usefulness of a ....

Voorhees, E., & Harman, D. (1997). Overview of the Fifth Text REtrieval Conference (TREC-5). In E. M. Voorhees & D. K. Harman (Eds.), The Fifth Text Retrieval Conference (TREC-5). (pp. 1-28). Gaithersburg, MD, USA.

....much more likely to agree correctly than to agree in error. This property, if true, can be used to improve precision by any fusion method which gives more weight to common documents than non common documents. In fact, it has been accepted as truth or taken for granted in some IR literature (e.g. Voorhees and Harman 1997). We call this line of reasoning the improved precision argument . In theory, using the improved precision argument for data fusion can be quite simple, or quite complicated. It depends on whether or not we consider the attributes (or decisions) assigned by different IR schemes to each document ....

Voorhees, E.M. & Harman, D. (1997). Overview of the Fifth Text REtrieval Conference. In D.

....to the document in which they occur. In particular they provide information on how terms are used within the document and will be used in our experiments to differentiate between documents. 3. Data In these experiments we used the Wall Street Journal (1990 92) WSJ) collection from TREC 5 (Voorhees and Harman, 1996) and the Financial Times (FT) collection from the TREC 6 (Voorhees and Harman, 1997) set of collections. The details of these collections are summarised in Table 1. Table 1: Details of collections used Collection FT WSJ Number of documents 204790 74580 Number of queries used 2 38 30 Average ....

Voorhees, E. M. and D. Harman, (1996). Overview of the Fifth Text REtrieval Conference (TREC-5). In: Proceedings of the 5th Text Retrieval Conference. Gaitherburg, MD. pp 129. Nist Special Publication 500-238.

....how many relevant documents a Querying Text Datasets p. 2 Final Report corpus contains. The standard technique for estimating this number is pooling : identifying relevant documents from among those returned by all IR systems involved in a comparison. This method is used by the TREC program (Voorhees and Harman (1997)) Our method is a principled alternative to this method that is well grounded in statistical theory, and, unlike pooling, is independent of any biases present in current IR systems. In either application type, the method is designed to address statistical questions that are: subjective: that ....

....the advantage of applying user judgments to representative documents from the entire corpus, not just to the documents found by current systems. Accordingly, for our second query we chose one from a nationwide information retrieval system evaluation, the TREC program sponsored by NIST (see e.g. Voorhees Harman 1997). The query selected was ad hoc query #83, from TREC 1: Measures to Protect the Atmosphere . The topic statement for this query is included as Appendix B. Along with the topic statement, we obtained from the TREC archives a qrels file for this query. The file listed all documents that at least ....

Voorhees E. and Harman D. (1997) Overview of the Fifth Text REtrieval Conference (TREC5) . In "Proceedings of the Fifth Text REtrieval Conference (TREC-5)", E. Voorhees & D.

....system actually finds. To establish recall, one must know how many relevant documents exist. The standard technique for estimating this number is pooling : identifying relevant documents from among those returned by all IR systems involved in a comparison. This method is used by the TREC program (Voorhees and Harman (1997)) Our method is a principled alternative to this method that is well grounded in statistical theory, and, unlike pooling, is independent of any biases present in current IR systems. Applying the method to a new question, whether for its own sake or to determine recall, involves developing a ....

Voorhees E. and Harman D. (1997) Overview of the Fifth Text REtrieval Conference (TREC-5). In "Proceedings of the Fifth Text REtrieval Conference (TREC-5)", E. Voorhees & D. Harman, ed., NIST Special Publication 500-238, pp. 1-28.

....of at least the order of hundreds of thousands of documents constituting gigabytes of text. Probably the greatest drive for this has come from TREC, an annual evaluation and benchmarking exercise coordinated annually by the National Institute for Standards and Technology (NIST) since 1992 [16] [8] In TREC, as in most IR research, the evaluation of effectiveness is measured in terms of the precision (percentage documents retrieved that are relevant) and recall (percentage relevant documents retrieved) averaged over a set of test queries. TREC has developed into a global coordinated ....

....representation of document images. 5 Word Shape Tokens for Information Retrieval We evaluated the retrieval effectiveness of WST based retrieval on two experimental collections of documents, queries and relevance assessments taken from the annual series of DARPA funded TREC benchmarking exercises [16]. The TREC environment provides a controlled environment where the relevance of documents to topics or queries are determined manually and these are then made available to the scientific community for subsequent information retrieval experiments like ours. To evaluate the usefulness of WST based ....

[Article contains additional citation context not shown here]

Voorhees E., Harman, D.H.: Overview of the Fifth Text Retrieval Conference (TREC-5). In: The Fifth Text Retrieval Conference (TREC-5). NIST Special Publication 500-238 (1997) 1-28

.... is a significant increase in retrieval effectiveness (for more details, see Sections 4 and 5) Recent experiments by several groups participating in TREC also suggest that, averaged over large query sets (with 50 queries) adhoc expansion yields significant improvements in overall performance [20, 1, 4, 11]. Thus, it seems worthwhile to continue using adhoc expansion in retrieval for short queries, while exploring ways to prevent query drift the alteration of the focus of a search topic caused by improper expansion (as described in the above example) Since the presence of a large proportion of ....

....investigated more extensively in the next section. 4 Experiments and Results In order to determine the usefulness of the techniques described above, we test them on a variety of tasks. We use the TREC collections in our experiments. Our methods are evaluated on the adhoc tasks for TRECs 3 6 [8, 9, 20, 21]. The query sets and document collections used in these tasks are shown in Table 1. Since we are interested in studying short queries, we use only the Description field for queries 151 300. For the TREC 6 queries (numbered 301 350) we use the Title field in addition 3 . Our experiments ....

E. M. Voorhees and D. K. Harman. Overview of the Fifth Text REtrieval Conference (TREC-5) . In E. M. Voorhees and D. K. Harman, editors, Proceedings of the Fifth Text REtrieval Conference (TREC-5). NIST Special Publication 500-238, 1997.

....top 5 documents are relevant and the 100 th document is non relevant. Variations on this method of expanding the initial query by pseudo relevance feedback (using terms from the top n documents) have been used by top performing participants in past TREC ad hoc experiments (Buckley et al. 1995; Voorhees Harman, 1997). In addition to query expansion by phrases and automatic feedback, we also tested query expansion methods by using passages 6 with matching initial query terms. Three variations of query expansion by passage feedback were tested by selecting terms from only 6 IRIS identifies a passage boundary ....

....at 2 or 3 of total collection cutoff will still be applicable in other instances. Though high recall value at such a low rank (under 3 of the total document collection) is somewhat suspect due to the potential bias introduced by the TREC pooling method of relevant document identification (Voorhees Harman, 1997), it is still reasonable to think that subcollection IR can be an effective as well as efficient way to deal with the problem of massive document collections. 4 Ad hoc Experiment 4.1 Research Question As a natural consequence of our belief that the user is an integral component of a truly ....

Voorhees, E., & Harman, D. (1997). Overview of the Fifth Text Retrieval Conference. In E. M. Voorhees & D. K. Harman (Eds.), The Fifth Text REtrieval Conference (TREC-5).

....approach and how our retrieval system operates. The next section presents results from our experiments. 4 Experiments 4. 1 Documents, Queries and Evaluation The documents we used in our experiments were taken from the TREC data set, specifically the set of documents used in category B of TREC 5 [10]. The queries and relevance judgments we used were derived from the topic statements for TREC 5 with an average of 21 relevant documents per topic. Evaluation is performed by calculating precision at the 11 standard recall points as used in TREC [10] In all the retrieval experiments we report ....

....set of documents used in category B of TREC 5 [10] The queries and relevance judgments we used were derived from the topic statements for TREC 5 with an average of 21 relevant documents per topic. Evaluation is performed by calculating precision at the 11 standard recall points as used in TREC [10]. In all the retrieval experiments we report here, retrieval is based on scoring each document in the collection based on the sum of the tf ThetaI DF weights of search terms occurring in each document and Figure 1: Schematic Outline of our WST based Retrieval System. ranking documents based on ....

E. Voorhees & D. Harman. Overview of the Fifth Text Retrieval Conference (TREC-5). In The Fifth Text Retrieval Conference (TREC-5), NIST Special Publication 500-238, pages 1--28, 1997.

No context found.

Voorhees, E. M. & Harman, D. (1997). Overview of the Fifth Text REtrieval conference (TREC-5). In Harman, D. (Ed.), TREC-5, Proceedings of the Fourth Text Retrieval Conference (pp.1-28). Washington, DC: Government Printing Office.

No context found.

Voorhees, E. M. and Harman, D. Overview of the Fifth Text REtrieval Conference (TREC-5). Proceedings of the 5th Text Retrieval Conference. 1-29. Nist Special Publication 500-238. Gaitherburg.1996.

No context found.

E. M. Voorhees and D. Harman. Overview of the Fifth Text REtrieval Conference

No context found.

E. M. Voorhees and D. Harman. Overview of the Fifth Text REtrieval Conference (TREC-5). Proceedings of the 5th Text Retrieval Conference. pp 1-29. Nist Special Publication 500-238. Gaitherburg.1996.

No context found.

Ellen Voorhees and Donna Harman. Overview of the fifth Text Retrieval Conference (TREC5) . In E. M. Voorhees and D. K. Harman, editors, Proceedings of TREC-5, pages 1--28, Gaithersburg MD, November 1996. NIST special publication 500-238, http://trec.nist.gov.

No context found.

Ellen M. Voorhees and Donna Harman. 1997. Overview of the fifth Text REtrieval Conference (TREC-5). In Proceedings of the Fifth Text REtrieval Conference (TREC-5), pages 1--28. NIST 500-238.

First 50 documents

Online articles have much greater impact   More about CiteSeer.IST   Add search form to your site   Submit documents   Feedback  

CiteSeer.IST - Copyright Penn State and NEC