Performance measures for information retrieval systems—an experimental approach

John J. Regazzi

Journal of the American Society for Information Science, 1988, vol. 39, issue 4, 235-251

Abstract: This study is an investigation into some critical factors which would constitute reliable performance measures for bibliographic information retrieval systems. Thirty‐two (32) judges, grouped by type (researcher or student), level (senior or junior), specialty (biomedicine or social science), and evaluation context (relevance or utility), were asked to rate 16 documents on alcohol studies which included four different search topics and four document citations and abstracts for each search topic. Half of the judges were asked to rate the documents on how relevant the documents were to the search topic; the other half rated the same documents on the basis of the document's perceived utility for the individual judge. The topics and documents were presented in a controlled prescribed order, so as to test the effects of learning on the evaluation process. After rating the documents, judges were also asked to rate the importance of five document attributes (author, title, abstract, source of publication, and date of publication), and six information attributes (accuracy, completeness, subject, suggestiveness, timeliness, and treatment). The study finds that there is no operational difference between the relevance‐theoretic and the utility‐theoretic model of evaluation. It further suggests the need for performance measures based upon a complex set of factors including document and information attributes, the judge, and other environmental factors, particularly the effects of learning during the evaluation process. © 1988 John Wiley & Sons, Inc.

Date: 1988
References: Add references at CitEc
Citations: View citations in EconPapers (1)

Downloads: (external link)
https://doi.org/10.1002/(SICI)1097-4571(198807)39:43.0.CO;2-H

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:bla:jamest:v:39:y:1988:i:4:p:235-251

Ordering information: This journal article can be ordered from
https://doi.org/10.1002/(ISSN)1097-4571

Access Statistics for this article

More articles in Journal of the American Society for Information Science from Association for Information Science & Technology
Bibliographic data for series maintained by Wiley Content Delivery ().