David Hawking, Nick Craswell, Paul Thistlewaite, and Donna Harman. Results and challenges in web search evaluation. Proceedings of WWW8, 31:1321--1330, 1999. http://www8.org/w8-papers/ 2c-search-discover/results/results.html.  Home/Search   Document Details and Download   Summary   Related Articles   Check

....the number of them does not. Thus burstsort and MSD radixsort have the same asymptotic computational cost as given earlier. 4 Experiments We have used three kinds of data in our experiments, words, genomic strings and web URLs. The words are drawn from the large web track in the TREC project [6, 7], and are alphabetic strings delimited by non alphabetic characters in web pages (after removal of tags, images, and other non text information) The web URLs have been drawn from the same collection. The genomic strings are from GenBank. For word and Some of these data sets are available under ....

D. Hawking, N. Craswell, P. Thistlewaite, and D. Harman. Results and challenges in web search evaluation. In Proc. World-Wide Web Conference, 1999.

....their efforts, the answers provided by these systems are generally not very satisfactory. Preliminary results obtained with a test collection of the TREC conference Web Track has showed the poor results quality of 5 well known engines of the Web, compared to those of 6 systems taking part to TREC [17]. In fact, most of existing search engines use well known techniques like those described by Salton 30 years ago [30] Most of them prefer a wide coverage of Web with a low indexing quality to a better indexing on a smaller part of the Web. In particular, they consider generally HTML pages as ....

D. Hawking, N. Craswell, P. Thistlewaite, and D. Harman. Results and challenges in Web search evaluation. In World Wide Web Conference (WWW'99), Toronto, Canada, May 1999.

....2,073,418,204 pages and process thousands of queries per second. These systems allow nding pages using criteria mainly regarding the textual content. Results obtained with a test collection of TREC Web track has showed the lower quality of 5 Web search engines compared to 6 TREC systems [3]. More recently, some of 20 Web search engines have showed quality improvement, one of them reaching the average precision of the median TREC system [4] In fact, these systems are based mainly on classic IR model, as proposed by Salton 30 years ago [5] documents are considered as atomics and ....

David Hawking, Nick Craswell, Paul Thistlewaite, and Donna Harman, \Results and challenges in Web search evaluation," in World Wide Web Conference, Toronto, Canada, May 1999.

....collection is so large, it is not possible to manually judge enough queries to a sufficient result depth to be able to measure recall in any reasonable way. Other researchers have also enumerated these fundamental challenges of web evaluations, however they have focused on the ad hoc search task [1]. While other researchers have examined the effectiveness of various search services, their primary evaluation technique has been some precision variant evaluation of informational queries [3, 4] Singhal et al. evaluated the search tasks of web users and proposed that navigational queries were ....

D. Hawking, N. Craswell, P. Thistlewaite, and D. Harman. "Results and challenges in web search evaluation. ", in Proc. Eighth Int'l World Wide Web Conf., pages 243-252, May 1999.

....the query should be directed to this zone. The chosen classification system is the ODP Web Directory [11] which consists of approx. 380 000 topics. 1. This zone is equivalent to a branch of the topic hierarchy We have employed an automatic classification of the TREC WT10g document collection [12] in order to study the recall 2 of a retrieval system based on the CSI design. The WT10g collection consists of 1.7 million documents fetched from the World Wide Web, 100 topics (information queries expressed in normal language) and relevance judgements which list the documents regarded as the ....

Hawking, D., et al., Results and Challenges in Web Search Evaluation, Computer Network, 31,1999, pp. 1321-1330.

....queries; 2word queries only; and 5 word queries only. words stopped time (sec) queries queries 0 1.56 0.49 6.41 3 0.66 0.30 1.94 6 0.45 0.29 1.07 10 0.40 0.28 0.81 20 0.37 0.28 0.70 254 0.18 0.16 0.26 is 21.9 Gb of HTML containing about 8. 3 Gb of text (drawn from the TREC large web track [9]) Table 1 shows the size of the index with a range of levels of stopping. As can be seen, the three commonest words account for around 4 of the index size, and only small space savings are yielded by stopping. However, as Table 2 shows, the impact of stopping on query evaluation time is ....

D. Hawking, N. Craswell, P. Thistlewaite, and D. Harman. Results and challenges in web search evaluation. In Proc. of the Eighth International World-Wide Web Conference, volume 31, pages 1321--1330, May 1999.

....not very satisfactory: they are often too many, disturbed, not very precise with a lot of noise. Preliminary results obtained with a test collection of the TREC Conference Web Track showed the poor results quality of 5 well known search engines, compared with those of 6 systems taking part in TREC [17]. Berghel considers the current search engines as primitive versions of the future means of information access on the Web [5] mainly because of their incapacity to distinguish the good from the bad in this huge amount of information. According to him, this evolution cannot be done by simple ....

....pages to put forward a relevance judgement. In this way, we know which are the p most relevant pages (R p ) for a query according to a judge. n documentsprecision = k R p k n (5) There do exist such test collections for the Web, created as part of the TREC (Text Retrieval Conference) Web Track [17]. Two samples of respectively 1 gigabyte (GB) and 10 GB were extracted from a 100 GB corpus of Web documents. A set of queries similar to those used in a more traditional context was used with these collections, with several TREC systems and ve well known search engines. Judges performed ....

Hawking, D., Craswell, N., Thistlewaite, P., Harman, D.: Results and Challenges in Web Search Evaluation. 8th World Wide Web Conference (WWW'99), Toronto, Canada, (1999)

....then would like to examine other similar documents. Search engines are still in their infancy. Although the exact nature of many of the algorithms they use for finding appropriate pages is proprietary [24] there is some research that suggests that the algorithms they use are not very accurate [15]. Existing search engines and their users are typically at cross purposes. While these systems normally retrieve documents based on low level features, users usually have a more abstract notion of what will satisfy them when conducting a query for certain information. For instance, most search ....

D. Hawking, Results and Challenges in Web Search Evaluation, Proceedings of the 8 th International World Wide Web Conference, Toronto, Canada, May 1999.

....the question by using the traditional IR approach of eliminating stopwords. To provide a greater potential for redundancy and to provide an opportunity to study the impact of database size on the results, we eschewed the 3GB TREC 9 QA corpus in favor a larger corpus, the TREC 100GB VLC2 corpus [12,21]. Apart from its size, this corpus di ers from the QA corpus in two important respects. First, since the QA corpus consists of newspaper articles and the VLC2 corpus of arbitrary Web pages, we might generally expect documents in the VLC2 corpus to be of lower quality. Second, it is known that ....

David Hawking, Nick Craswell, Paul Thistlewaite, and Donna Harman. Results and challenges in Web search evaluation. In 8th World Wide Web Conference, pages 243-252, Toronto, Canada, May 1999.

....the MultiText [7] structured text database system that has been used to support large digital library and WWW search applications. Using a specialized index and adaptive algorithms, simple queries can be executed in under a second over 100 GB of data. As a result of using a large capacity database [20], Jupiter avoids any database related scalability issues [6] While Jupiter is usable on its own as a program comprehension tool it can also function as a backend for other tools. For example, MARS [13] the predecessor to Jupiter, was used to extend the Software Bookshelf [18] with search ....

D. Hawking, N. Craswell, P. Thistlewaite, and D. Harman. Results and challenges in web search evaluation. In Proceedings of the Eighth World Wide Web Conference, pages 243--252, Toronto, May 1999.

.... link information in some form [1, 25] Howmuch more e ective are link based methods in the web environment as compared to a state of the art keywordbased method developed for the TREC ad hoc task This question has been studied in a limited number of studies, especially under TREC s web track [5, 6, 7]. The results from these studies indicate that for web search, link based methods do not hold anyadvantage over the state of the art keyword based methods developed for TREC ad hoc search. These results are quite counter intuitive given the general wisdom in the web search community that some kind ....

....keyword based methods developed for TREC ad hoc search. These results are quite counter intuitive given the general wisdom in the web search community that some kind of linkage analysis does improveweb page site ranking. Our work is motivated by this discrepancy between the results presented in [5, 6, 7], and the general belief in the web search community. Di erentweb search engines make competing claims regarding their coverage and search e ectiveness. In this study, we don t concentrate on comparing the search e ectiveness of di erent web search engines. There have been several studies that ....

[Article contains additional citation context not shown here]

D. Hawking, N. Craswell, P. Thistlewaite, and D. Harman. Results and challenges in web search evaluation. In ########### ## ### ###### ############# ##### #### ### ##########, pages 243-252, May 1999.

....terms encountered may not be present in the training collection. We use IDF (Inverse Document Frequency, which is high for rare terms, and low for common terms) weights derived from a much larger sample (one million web pages, obtained from the collection of pages used in the TREC Web Track [9]) The last, fall back case is necessary in order to handle phrases not present in the training data. Intuitively, it assigns the weight of phrase # inversely proportional to the probability that all the terms in # appear together, scaled to weight occurrences of multi word phrases higher. This ....

....for the web, which contains billions of documents. Researchers have addressed this problem by developing standard document collections with queries and associated relevance judgments, and by limiting the domain of documents that are judged [21] Recently, TREC has incorporated a Web Track [9] that employs a collection of web documents (small relative to the size of the web) This collection of documents is a valuable resource to evaluate information retrieval algorithms over web data. However, the collection is not well suited to evaluate a system like Tritus,where we aim to ....

D. Hawking, N. Craswell, P. Thistlewaite, and D. Harman. Results and challenges in web search evaluation. In 8th International World Wide Web Conference (WWW-8), 1999.

....253.24 Table 5: Some of the top ranked transforms for the question phrase what is a automatically optimized for AltaVista and Google. and low for common terms) weights derived from a much larger sample (one million web pages, obtained from the collection of pages used in the TREC Web Track [9]) The last, fall back case is necessary in order to handle phrases not present in the training data. Intuitively, it assigns the weight of phrase t inversely proportional to the probability that all the terms in t appear together, scaled to weight occurrences of multi word phrases higher. This ....

....for the web, which contains billions of documents. Researchers have addressed this problem by developing standard document collections with queries and associated relevance judgments, and by limiting the domain of documents that are judged [21] Recently, TREC has incorporated a Web Track [9] that employs a collection of web documents (small relative to the size of the web) This collection of documents is a valuable resource to evaluate information retrieval algorithms over web data. However, the collection is not well suited to evaluate a system like Tritus, where we aim to ....

D. Hawking, N. Craswell, P. Thistlewaite, and D. Harman. Results and challenges in web search evaluation. In 8th International World Wide Web Conference (WWW-8), 1999.

....column holds the proportion of unrelated documents. We averaged these individual proportions to get a precision indicator of 23 for documents relevant to intended concept. While this may seem low, it is understandable in view of the fact that Internet search engines have a precision of 23 to 38 (Hawking, et al. 1999). Moreover, some documents, although irrelevant for the target concept, may still be useful for some other concepts in the structure. Note that the simple strategy of formulating queries by extracting terms from the concept descriptor is generally effective; there are notable exceptions, however. ....

Hawking, D., Craswell, N., Thistlewaite, P. and Harman, D. (1999). Results and challenges in Web search evaluation, in Proceedings of the Eighth International World Wide Web Conference(pp. 1321-1330), North Holland.

....search engines, such as: single keyword search; plural search capability; phrase search; Boolean search (with proper noun) and complex Boolean. In the next section, we discuss some of the di erences and similarities in classical and Internet based search, access and retrieval of information. [Hawking et al. 1999] discusses evaluation studies of six TREC 34 search engines. In particular, they examine answers to questions, such as: Can link information result in better rankings and Do longer queries result in better answers . 34 U.S. National Institute of Standards and Technology (NIST) Text ....

Hawking, D., Craswell, N., Thistlewaite, P., Harman, D., \Results and challenges in Web search evaluation", Computer Networks (special issue: Proc. 8th World Wide Web Conference), 31 (1999), 1321-1330.

....contained, were evaluated, and duplicate documents were treated as ordinary ones. Documents that could not be evaluated due to no response , le not found etc. were simply passed over and a new document was selected instead. A similar relevance judgement procedure was used by Hawking et al. [8] in a comparison between some TREC IR systems and some well known Web search engines. Using the described procedure, the P 20 values were estimated to 0.33 for the o cial version of the engine, and to 0.36 for the new version. Hawking et al. 8] found a precision range of 0.23 to 0.38 for the ....

....judgement procedure was used by Hawking et al. 8] in a comparison between some TREC IR systems and some well known Web search engines. Using the described procedure, the P 20 values were estimated to 0.33 for the o cial version of the engine, and to 0.36 for the new version. Hawking et al. [8] found a precision range of 0.23 to 0.38 for the Web search engines they tested. Is an observed precision di erence of 0.03 signi cant A standard paired data two tailed test with the null hypothesis that the di erence is zero, is not rejected at the 0.05 level of signi cance (z=1.79 against z ....

[Article contains additional citation context not shown here]

D. Hawking, N. Craswell, P. Thistlewaite, and D. Harman. Results and challenges in Web search evaluation. http://pastime.anu.edu.au/TAR/www8.ps.gz, (as visited June, 17th, 1999), 1999.

No context found.

David Hawking, Nick Craswell, Paul Thistlewaite, and Donna Harman. Results and challenges in web search evaluation. Proceedings of WWW8, 31:1321--1330, 1999. http://www8.org/w8-papers/ 2c-search-discover/results/results.html.

No context found.

David Hawking, Nick Craswell, Paul Thistlewaite, and Donna Harman. Results and challenges in web search evaluation. Proceedings of WWW8, 31:1321--1330, 1999. http://www8.org/w8-papers/ 2c-search-discover/results/results.html.

....search engines facilitates informed consumer choice and may also lead to a general raising of search quality. As well as various informal and subjective comparisons in the media, a number of academic studies have attempted to compare the performance of search engines on a scientific basis. [2, 3, 4, 5, 6] Most Information Retrieval experiments and all of the published scientific evaluations of commercial search engines have measured the ability of search systems to retrieve documents which are relevant to a topic of interest. However, queries submitted to commercial search engines reflect a range ....

David Hawking, Nick Craswell, Paul Thistlewaite, and Donna Harman. Results and challenges in web search evaluation. Proceedings of WWW8, 31:1321--1330, 1999. http://www8.org/w8-papers/ 2c-search-discover/results/results.html.

....performing boolean unranked retrieval. Based on statistics alone these servers are identical, but the former is almost certainly a better selection than the latter in practice. In an environment such as the Web where many different retrieval systems are used and retrieval effectiveness varies [12], we hypothesize that estimating server effectiveness is important. Our initial method for estimating server retrieval effectiveness is unrefined (results later in the paper indicate it requires further study and tuning) We choose a broker specific point in the ranking ten documents, since ....

David Hawking, Nick Craswell, Paul Thistlewaite, and Donna Harman. Results and challenges in Web search evaluation. In Proceedings of WWW8, Toronto, pages 243--252. Elsevier, 1999. http: //pastime.anu.edu.au/nick/pubs/www8.pdf.

....1 Introduction Experiments using TREC style topic descriptions and relevance judgments have recently been carried out for the rst time over real Web data. One interesting result is that systems of TREC VLC Track participants [5] were more e ective than live Web systems [6]. A number of factors could explain this, but one important possibility is that Web and TREC ad hoc systems are solving di erent problems. If this were the case, it would be unfair to use TREC ad hoc evaluation methods as a benchmark for the success of Web retrieval systems. Here we informally ....

....on the basis of early precision. The measure used was precision at 20 documents retrieved which is de ned as the proportion of the top twenty documents retrieved which were judged relevant. At the same time as assessors were judging VLC track documents, results from ve popular live Web systems [6] were presented to them for judgment. Live systems each index di erent and overlapping sets of Web documents, rather than the 100 gigabyte VLC. However, results were judged in the same way. A summary of a subset of the e ectiveness results obtained is presented in Table 1. Please see [6] for full ....

[Article contains additional citation context not shown here]

David Hawking, Nick Craswell, Paul Thistlewaite, and Donna Harman. Results and challenges in web search evaluation. In Proceedings of WWW8, Toronto, pages 243-252. Elsevier, 1999.

No context found.

D. Hawking, N. Craswell, P. B. Thistlewaite, and D. Harman. Results and challenges in web search evaluation. Computer Networks, 31(11--16):1321--1330, May 1999. Also in Proceedings of the 8th International World Wide Web Conference.

No context found.

D. Hawking, N. Craswell, P. B. Thistlewaite, and D. Harman. Results and challenges in web search evaluation. Computer Networks, 31(11--16):1321--1330, May 1999. Also in Proceedings of the 8th International World Wide Web Conference.

No context found.

Hawking, D., Craswell, N., Thistlewaite P., and Harman, D. Results and challenges in web search evaluation. In Proceedings of WWW8 (Toronto, Canada, May 1999.

No context found.

David Hawking, Nick Craswell, Paul Thistlewaite, and Donna Harman. Results and challenges in web search evaluation. In Proceedings of the Eighth International World Wide Web Conference, http://www8.org/w8-papers/2c-search-discover/ results/results.html, 1999.

Online articles have much greater impact   More about CiteSeer.IST   Add search form to your site   Submit documents   Feedback  

CiteSeer.IST - Copyright Penn State and NEC