DMCA
Probabilistic Models for Information Retrieval based on Divergence from Randomness (2002)
Cached
Download Links
| Venue: | ACM TRANSACTIONS ON INFORMATION SYSTEMS |
| Citations: | 244 - 5 self |
BibTeX
@ARTICLE{Amati02probabilisticmodels,
author = {Gianni Amati and Cornelis Joost Van Rijsbergen},
title = {Probabilistic Models for Information Retrieval based on Divergence from Randomness},
journal = {ACM TRANSACTIONS ON INFORMATION SYSTEMS},
year = {2002},
volume = {20},
pages = {357--389}
}
Share
 |
 |
 |
 |
||
OpenURL
Abstract
We introduce and create a framework for deriving probabilistic models of Information Retrieval. The models are nonparametric models of IR obtained in the language model approach. We derive term-weighting models by measuring the divergence of the actual term distribution from that obtained under a random process. Among the random processes we study the binomial distribution and Bose–Einstein statistics. We define two types of term frequency normalization for tuning term weights in the document–query matching process. The first normalization assumes that documents have the same length and measures the information gain with the observed term once it has been accepted as a good descriptor of the observed document. The second normalization is related to the document length and to other statistics. These two normalization methods are applied to the basic models in succession to obtain weighting formulae. Results show that our framework produces different nonparametric models forming baseline alternatives to the standard tf-idf model.
Keyphrases
information retrieval probabilistic model baseline alternative derive term-weighting model nonparametric model term weight second normalization good descriptor normalization method basic model observed term observed document random process bose einstein statistic term frequency normalization document length first normalization weighting formula binomial distribution document query actual term distribution standard tf-idf model language model approach different nonparametric model information gain
