Fuzzy versus probabilistic models for user relevance judgments

Mathew Koll and Padmini Srinivasan

Journal of the American Society for Information Science, 1990, vol. 41, issue 4, 264-271

Abstract: A crucial aspect of information retrieval is the process of making relevance judgments. Although this highly complex decision making procedure still eludes researchers, it is evident that a number of mental models are involved: models of the information need, retrieval system, database, user's knowledge in the subject area etc. In general, it is accepted that relevance judgments are made by evaluating documents for an overall conceptual match with the information need. In this study, we take the view that, given a document and a query, users first judge the document against the individual concepts in the query and then use some inferencing process to derive from these “atomic” decisions a “compound” judgment for the entire query. In other words we adopt a bottom up approach to this decision making process. In this context we examine strategies that may be used to infer compound relevance judgments made from judgments against smaller units of the information need. The probabilistic and fuzzy models are used as two points of reference against which to analyze the user's decision process in making compound judgments. Each model has a different way of approaching the problem. The objective was to empirically examine the relative effectiveness of the models at predicting compound judgments made by users, from their atomic judgments. The results show that the fuzzy model is sometimes a better predictor than the probabilistic model. The conclusions are interesting when comparing performances across AND and OR queries. No differences were observed between the two models for the OR queries. Both models tend to underestimate relevance for the AND queries. However, the probabilistic model regularly underestimated relevance more than the fuzzy model. The conclusion made is that the user seldom employs the AND operator as rigidly as assumed by these models and perhaps by most IR systems. Also, matching functions that better approximate the user's decision process are required to lead to more effective systems. © 1990 John Wiley & Sons, Inc.

Date: 1990
References: Add references at CitEc
Citations:

Downloads: (external link)
https://doi.org/10.1002/(SICI)1097-4571(199006)41:43.0.CO;2-3

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:bla:jamest:v:41:y:1990:i:4:p:264-271

Ordering information: This journal article can be ordered from
https://doi.org/10.1002/(ISSN)1097-4571

Access Statistics for this article

More articles in Journal of the American Society for Information Science from Association for Information Science & Technology
Bibliographic data for series maintained by Wiley Content Delivery ().