An experimental comparison of click position-bias models

An experimental comparison of click position-bias models (2008)

by Nick Craswell, Onno Zoeter, Michael Taylor, Bill Ramsey

Venue:

In WSDM ’08: Proceedings of the First ACM International Conference on Web Search and Data Mining

Add To MetaCart

Results 1 - 10 of 193

Expected Reciprocal Rank for Graded Relevance

by Olivier Chapelle, Ya Zhang - CIKM'09, NOVEMBER 2–6, 2009, HONG KONG, CHINA. , 2009

"... While numerous metrics for information retrieval are available in the case of binary relevance, there is only one commonly used metric for graded relevance, namely the Discounted Cumulative Gain (DCG). A drawback of DCG is its additive nature and the underlying independence assumption: a document in ..."

Abstract - Cited by 158 (12 self) - Add to MetaCart

While numerous metrics for information retrieval are available in the case of binary relevance, there is only one commonly used metric for graded relevance, namely the Discounted Cumulative Gain (DCG). A drawback of DCG is its additive nature and the underlying independence assumption: a document in a given position has always the same gain and discount independently of the documents shown above it. Inspired by the “cascade ” user model, we present a new editorial metric for graded relevance which overcomes this difficulty and implicitly discounts documents which are shown below very relevant documents. More precisely, this new metric is defined as the expected reciprocal length of time that the user will take to find a relevant document. This can be seen as an extension of the classical reciprocal rank to the graded relevance case and we call this metric Expected Reciprocal Rank (ERR). We conduct an extensive evaluation on the query logs of a commercial search engine and show that ERR correlates better with clicks metrics than other editorial metrics.

Yahoo! Learning to Rank Challenge Overview

by Olivier Chapelle, Yi Chang , 2011

"... Learning to rank for information retrieval has gained a lot of interest in the recent years but there is a lack for large real-world datasets to benchmark algorithms. That led us to publicly release two datasets used internally at Yahoo! for learning the web search ranking function. To promote these ..."

Abstract - Cited by 72 (6 self) - Add to MetaCart

Learning to rank for information retrieval has gained a lot of interest in the recent years but there is a lack for large real-world datasets to benchmark algorithms. That led us to publicly release two datasets used internally at Yahoo! for learning the web search ranking function. To promote these datasets and foster the development of state-of-the-art learning to rank algorithms, we organized the Yahoo! Learning to Rank Challenge in spring 2010. This paper provides an overview and an analysis of this challenge, along with a detailed description of the released datasets.

Efficient multiple-click models in web search

by Fan Guo, Chao Liu, Yi-min Wang , 2008

"... Many tasks that leverage web search users’ implicit feedback rely on a proper and unbiased interpretation of user clicks. Previous eye-tracking experiments and studies on explaining position-bias of user clicks provide a spectrum of hypotheses and models on how an average user examines and possibly ..."

Abstract - Cited by 59 (3 self) - Add to MetaCart

Many tasks that leverage web search users’ implicit feedback rely on a proper and unbiased interpretation of user clicks. Previous eye-tracking experiments and studies on explaining position-bias of user clicks provide a spectrum of hypotheses and models on how an average user examines and possibly clicks web documents returned by a search engine with respect to the submitted query. In this paper, we attempt to close the gap between previous work, which studied how to model a single click, and the reality that multiple clicks on web documents in a single result page are not uncommon. Specifically, we present two multiple-click models: the independent click model (ICM) which is reformulated from previous work, and the dependent click model (DCM) which takes into consideration dependencies between multiple clicks. Both models can be efficiently learned with linear time and space complexities. More importantly, they can be incrementally updated as new click logs flow in. These are well-demanded properties in reality. We systematically evaluate the two models on click logs obtained in July 2008 from a major commercial search engine. The data set, after preprocessing, contains over 110 thousand distinct queries and 8.8 million query sessions. Extensive experimental studies demonstrate the gain of modeling multiple clicks and their dependencies. Finally, we note that since our experimental setup does not rely on tweaking search result rankings, it can be easily adopted by future studies.

(Show Context)

A cascade model for externalities in sponsored search

by David Kempe, Mohammad Mahdian - In ACM EC-08 Workshop on Ad Auctions , 2008

"... Abstract. One of the most important yet insufficiently studied issues in online advertising is the externality effect among ads: the value of an ad impression on a page is affected not just by the location that the ad is placed in, but also by the set of other ads displayed on the page. For instance ..."

Abstract - Cited by 57 (1 self) - Add to MetaCart

Abstract. One of the most important yet insufficiently studied issues in online advertising is the externality effect among ads: the value of an ad impression on a page is affected not just by the location that the ad is placed in, but also by the set of other ads displayed on the page. For instance, a high quality competing ad can detract users from another ad, while a low quality ad could cause the viewer to abandon the page altogether. In this paper, we propose and analyze a model for externalities in sponsored search ads. Our model is based on the assumption that users will visually scan the list of ads from the top to the bottom. After each ad, they make independent random decisions with ad-specific probabilities on whether to continue scanning. We then generalize the model in two ways: allowing for multiple separate blocks of ads, and allowing click probabilities to explicitly depend on ad positions as well. For the most basic model, we present a polynomial-time incentive-compatible auction mechanism for allocating and pricing ad slots. For the generalizations, we give approximation algorithms for the allocation of ads. 1

(Show Context)

Generalized distances between rankings

by Ravi Kumar, Sergei Vassilvitskii - Proc. 19th International Conference on WorldWideWeb , 2010

"... Spearman’s footrule and Kendall’s tau are two well established distances between rankings. They, however, fail to take into account concepts crucial to evaluating a result set in information retrieval: element relevance and positional information. That is, changing the rank of a highly-relevant docu ..."

Abstract - Cited by 40 (0 self) - Add to MetaCart

Spearman’s footrule and Kendall’s tau are two well established distances between rankings. They, however, fail to take into account concepts crucial to evaluating a result set in information retrieval: element relevance and positional information. That is, changing the rank of a highly-relevant document should result in a higher penalty than changing the rank of an irrelevant document; a similar logic holds for the top versus the bottom of the result ordering. In this work, we extend both of these metrics to those with position and element weights, and show that a variant of the Diaconis–Graham inequality still holds — the generalized two measures remain within a constant factor of each other for all permutations. We continue by extending the element weights into a distance metric between elements. For example, in search evaluation, swapping the order of two nearly duplicate results should result in little penalty, even if these two are highly relevant and appear at the top of the list. We extend the distance measures to this more general case and show that they remain within a constant factor of each other. We conclude by conducting simple experiments on web search data with the proposed measures. Our experiments show that the weighted generalizations are more robust and consistent with each other than their unweighted counterparts.

(Show Context)

A model to estimate intrinsic document relevance from the clickthrough logs of a web search engine

by Georges Dupret, Ciya Liao - In Proc. WSDM , 2010

"... We propose a new model to interpret the clickthrough logs of a web search engine. This model is based on explicit assumptions on the user behavior. In particular, we draw conclusions on a document relevance by observing the user behavior after he examined the document and not based on whether a user ..."

Abstract - Cited by 39 (2 self) - Add to MetaCart

We propose a new model to interpret the clickthrough logs of a web search engine. This model is based on explicit assumptions on the user behavior. In particular, we draw conclusions on a document relevance by observing the user behavior after he examined the document and not based on whether a user clicks or not a document url. This results in a model based on intrinsic relevance, as opposed to perceived relevance. We use the model to predict document relevance and then use this as feature for a “Learning to Rank ” machine learning algorithm. Comparing the ranking functions obtained by training the algorithm with and without the new feature we observe surprisingly good results. This is particularly notable given that the baseline we use is the heavily optimized ranking function of a leading commercial search engine. A deeper analysis shows that the new feature is particularly helpful for non navigational queries and queries with a large abandonment rate or a large average number of queries per session. This is important because these types of query is considered to be the most difficult to solve.

(Show Context)

No Clicks, No Problem: Using Cursor Movements to Understand and Improve Search

by Jeff Huang, Ryen W. White, Susan Dumais

"... Understanding how people interact with search engines is important in improving search quality. Web search engines typically analyze queries and clicked results, but these actions provide limited signals regarding search interaction. Laboratory studies often use richer methods such as gaze tracking, ..."

Abstract - Cited by 37 (11 self) - Add to MetaCart

Understanding how people interact with search engines is important in improving search quality. Web search engines typically analyze queries and clicked results, but these actions provide limited signals regarding search interaction. Laboratory studies often use richer methods such as gaze tracking, but this is impractical at Web scale. In this paper, we examine mouse cursor behavior on search engine results pages (SERPs), including not only clicks but also cursor movements and hovers over different page regions. We: (i) report an eye-tracking study showing that cursor position is closely related to eye gaze, especially on SERPs; (ii) present a scalable approach to capture cursor movements, and an analysis of search result examination behavior evident in these large-scale cursor data; and (iii) describe two applications (estimating search result relevance and distinguishing good from bad abandonment) that demonstrate the value of capturing cursor data. Our findings help us better understand how searchers use cursors on SERPs and can help design more effective search systems. Our scalable cursor tracking method may also be useful in non-search settings.

(Show Context)

Classification-enhanced ranking

by Paul N. Bennett, Krysta Svore, Susan T. Dumais , 2010

"... Many have speculated that classifying web pages can improve a search engine’s ranking of results. Intuitively results should be more relevant when they match the class of a query. We present a simple framework for classification-enhanced ranking that uses clicks in combination with the classificatio ..."

Abstract - Cited by 36 (13 self) - Add to MetaCart

Many have speculated that classifying web pages can improve a search engine’s ranking of results. Intuitively results should be more relevant when they match the class of a query. We present a simple framework for classification-enhanced ranking that uses clicks in combination with the classification of web pages to derive a class distribution for the query. We then go on to define a variety of features that capture the match between the class distributions of a web page and a query, the ambiguity of a query, and the coverage of a retrieved result relative to a query’s set of classes. Experimental results demonstrate that a ranker learned with these features significantly improves ranking over a competitive baseline. Furthermore, our methodology is agnostic with respect to the classification space and can be used to derive query classes for a variety of different taxonomies.

What Makes them Click: Empirical Analysis of Consumer Demand for Search Advertising ∗

by Przemyslaw Jeziorski, Ilya Segal, Pricewaterhousecoopers Llp , 2010

"... We study users ’ response to sponsored-search advertising using data from Microsoft’s Live AdCenter distributed in the “Beyond Search ” initiative. We estimate a structural model of utility maximizing users, which quantifies “user experience ” based on their “revealed preferences, ” and predicts use ..."

Abstract - Cited by 31 (0 self) - Add to MetaCart

We study users ’ response to sponsored-search advertising using data from Microsoft’s Live AdCenter distributed in the “Beyond Search ” initiative. We estimate a structural model of utility maximizing users, which quantifies “user experience ” based on their “revealed preferences, ” and predicts user responses to counterfactual ad placements. In the model, each user chooses clicks sequentially to maximize his expected utility under incomplete information about the relevance of ads. We estimate the substitutability of ads in users ’ utility function, the fixed effects of different ads and positions, user uncertainty about ads ’ relevance, and user heterogeneity. We find substantial substitutability of ads, which generates large negative externalities: 40 % more clicks would occur in a hypothetical world in which each ad faces no competition. As for counterfactual ad placements, our simulations indicate that CTR-optimal matching increases CTR by 10.1% while user-optimal matching increases user welfare by 13.3%. Moreover, targeting ad placement to specific users could raise user welfare by 59%. Here, we find a significant suboptimality (up to 16 % of total welfare) in case the search engine tries to implement a sophisticated matching policy using a misspecified model that does not account for externalities. Finally, user welfare could be raised by 14 % if they had full information about the relevance of ads to them. The authors are grateful to Microsoft Corp. for providing the data and computing facilities and hosting them during the summer of 2008. The second author also acknowledges the support of the Toulouse Network for Information