The Monte Carlo method and the evaluation of retrieval system performance

Robert Burgin

Journal of the American Society for Information Science, 1999, vol. 50, issue 2, 181-191

Abstract: The ability to distinguish between acceptable and unacceptable levels of retrieval performance and the ability to distinguish between significant and non‐significant differences between retrieval results are important to traditional information retrieval experiments. The Monte Carlo method is shown to represent an attractive alternative to the hypergeometric model for identifying the levels at which random retrieval performance is exceeded in retrieval test collections and for overcoming some of the limitations of the hypergeometric model. The Monte Carlo method produces low performance thresholds for the individual test collections that are very similar to the thresholds derived by the hypergeometric model, both at the test collection level and at the individual query level. In addition, the Monte Carlo method is much less computer‐intensive than the hypergeometric model, can be used with measures of retrieval effectiveness that take the rank order of the retrieved documents into consideration, can be used to derive the probability of obtained results, and can be used to determine the statistical significance of difference between two or more retrieval results. The ability to use the Monte Carlo method to derive the probability of obtained results and to compare two or more retrieval results makes it possible to determine more accurately how well retrieval systems operate under specific conditions and, in conjunction with the presentation of individual query results, makes it possible to determine whether relationships between query characteristics and retrieval system performance exist. Understanding these relationships should lead to improvements in the effectiveness of retrieval systems.

Date: 1999
References: Add references at CitEc
Citations:

Downloads: (external link)
https://doi.org/10.1002/(SICI)1097-4571(1999)50:23.0.CO;2-9

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:bla:jamest:v:50:y:1999:i:2:p:181-191

Ordering information: This journal article can be ordered from
https://doi.org/10.1002/(ISSN)1097-4571

Access Statistics for this article

More articles in Journal of the American Society for Information Science from Association for Information Science & Technology
Bibliographic data for series maintained by Wiley Content Delivery ().