Abstract not found

Users ’ past search behaviour provides a rich context that an information retrieval system can use to tailor its search results to suit an individual’s or a community’s information needs. In this paper, we present an investigation of the variability in search behaviours for the same queries in a close-knit community. By examining web proxy cache logs over a period of nine months, we extracted a set of 135 queries that had been issued by at least ten users. Our analysis indicates that, overall, users clicked on highly ranked and relevant pages, but they tend to click on different sets of pages. Examination of the query reformulation history revealed that users often have different search intents behind the same query. We identify three major causes for the community’s interaction behaviour differences: the variance of task, the different intents expressed with the query, and the snippet and characteristics of retrieved documents. Based on our observations, we identify opportunities to improve the design of different search and delivery tools to better support community and individual search experience.

Many current evaluation techniques for information retrieval, such as test collections and simulations, are difficult to apply in situations where queries and preferred results are context-dependent. This is particularly true in personal metasearch applications, which provide a person with unified search access to all their usual online sources. A recently-proposed technique, based on presenting two or more search results sets in a single comparison interface, offers an alternative. We have embedded this technique in a working personal metasearch tool which we have distributed to volunteers. Initial experiments with server selection suggest that the technique is accepted by users, can operate over diverse and unarticulated contexts, and that the data it provides can provide a useful comparison to that from test collections. Further experimentation with the technique is continuing. Themes: Case studies, field experiments, simulations, etc. 1.

Abstract Improving the quality of search engine results is the goal of costly efforts by major Web search engine companies. Using in situ side-by-side result set comparisons and random assignment of brandnames to result sets, we investigated whether perceptions of quality were influenced by brand association. In the first experiment (15 searchers) we found no significant preference for or against results labelled “Google ” relative to those labelled “Yahoo!”. In the second experiment (20 searchers) result sets were again generated by Google and Yahoo! but were randomly labelled “Yahoo!” or “WebKumara ” (a fictitious name). Again, we found no significant preference for one brandname label over the other. Contrary to previous findings, we found a statistically significant preference for Googlegenerated results over those of Yahoo! when data from three separate experiments (total 70 subjects) was combined.

We introduce a method for evaluating the relevance of all visible components of a Web search results page, in the context of that results page. Contrary to Cranfield-style evaluation methods, our approach recognizes that a user‟s initial search interaction is with the result page produced by a search system, not the landing pages linked from it. Our key contribution is that the method allows us to investigate aspects of component relevance that are difficult or impossible to judge in isolation. Such contextual aspects include component-level information redundancy and cross-component coherence. We report on how the method complements traditional document relevance measurement and its support for comparative relevance assessment across multiple search engines. We also study possible issues with applying the method, including brand presentation effects, inter-judge agreement, and comparisons with document-based relevance judgments. Our findings show this is a useful method for evaluating the dominant user experience in interacting with search systems.

Abstract. Aggregated search is the task of incorporating results from different specialized search services, or verticals, into Web search results. While most prior work focuses on deciding which verticals to present, the task of deciding where in the Web results to embed the vertical results has received less attention. We propose a methodology for evaluating an aggregated set of results. Our method elicits a relatively small number of human judgements for a given query and then uses these to facilitate a metric-based evaluation of any possible presentation for the query. An extensive user study with 13 verticals confirms that, when users prefer one presentation of results over another, our metric agrees with the stated preference. By using Amazon’s Mechanical Turk, we show that reliable assessments can be obtained quickly and inexpensively. 1

The CSIRO ICT Centre has recently constructed a question answering (QA) system – My Instant Expert ™ – designed for mobile phones. The client-server system supports asking open domain natural language questions and attempts to find answers from the (English) Wikipedia. Due to the small display of mobile phone devices, the space

Abstract Web spam potentially causes three deleterious effects: unnecessary work for crawlers and search engines; diversion of traffic away from legitimate businesses; and annoyance to search engine users through poorer results. Past research on web spam has focused on spamming techniques, spam suppression techniques, and methods for classifying web content as spam or non-spam. Here we focus on the deterioration of search result quality caused by the presence of spam in a countryscale web. We present a framework for measuring the degradation in quality of search results caused by the presence of web spam. We index the 80 million page UK2006 web spam collection on one machine. We trial the proposed framework in an experiment with the UK2006 collection and demonstrate that simple removal of spam pages from result sets can increase result quality. We conclude that the framework is a reasonable vehicle for research in this area and outline changes necessary for planned future experiments.

Today, if a website owner or blogger wants to provide a search interface on their web site, they have essentially two options: web search or site search. Site search is often too narrow and web search often too broad. We propose a context-specific alternative: the use of ‘topical search engines ’ (TopS) providing results focused on a specific topic determined by the site owner. For example a photography blog could offer a search interface focused on photography. In this paper, we describe a promising new approach to easily create such topical search engines with minimal manual effort. In our approach, whenever we have enough contextual information, we alter ambiguous topic related queries issued to a generic search engine by adding contextual keywords derived from (topic-specific) query logs; the altered queries help focus the

or send email to: Technical-DOT-Reports-AT-cs-DOT-anu.edu.au A list of technical reports, including some abstracts and copies of some full reports may be found at: