A utility theoretic examination of the probability ranking principle in information retrieval

Michael D. Gordon and Peter Lenk

Journal of the American Society for Information Science, 1991, vol. 42, issue 10, 703-714

Abstract: We challenge the probability ranking principle in information retrieval from the perspectives of (1) signal detection‐decision theory and (2) utility theory. If three conditions are not met by an IR system that is producing predictive probabilities of relevance, then inquirers may incur costs that are too great by selecting first those documents that the system predicts have the highest probabilities of relevance. These three conditions are that predictive probabilities are well calibrated (predictively accurate); that they are reported with certainty; and that an inquirer independently assesses the relevance of all documents he or she retrieves. When these conditions are met, signal detection analysis with fixed decision‐theoretic costs shows that the probability ranking principle is advisable. More generally, retrieval in adherence with the probability ranking principle is also advisable even when utility‐theoretic costs (or benefits) that vary with the number of relevant documents retrieved are associated with retrieval. Specifically, we prove that the utility an inquirer receives from the relevant documents he or she retrieves is maximized by selecting those documents with the largest predictive probabilities of relevance. © 1991 John Wiley & Sons, Inc.

Date: 1991
References: Add references at CitEc
Citations: View citations in EconPapers (1)

Downloads: (external link)
https://doi.org/10.1002/(SICI)1097-4571(199112)42:103.0.CO;2-1

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:bla:jamest:v:42:y:1991:i:10:p:703-714

Ordering information: This journal article can be ordered from
https://doi.org/10.1002/(ISSN)1097-4571

Access Statistics for this article

More articles in Journal of the American Society for Information Science from Association for Information Science & Technology
Bibliographic data for series maintained by Wiley Content Delivery ().