User perspectives on relevance criteria: A comparison among relevant, partially relevant, and not‐relevant judgments

Kelly L. Maglaughlin and Diane H. Sonnenwald

Journal of the American Society for Information Science and Technology, 2002, vol. 53, issue 5, 327-342

Abstract: This study investigates the use of criteria to assess relevant, partially relevant, and not‐relevant documents. Study participants identified passages within 20 document representations that they used to make relevance judgments; judged each document representation as a whole to be relevant, partially relevant, or not relevant to their information need; and explained their decisions in an interview. Analysis revealed 29 criteria, discussed positively and negatively, that were used by the participants when selecting passages that contributed or detracted from a document's relevance. These criteria can be grouped into six categories: abstract (e.g., citability, informativeness), author (e.g., novelty, discipline, affiliation, perceived status), content (e.g., accuracy/validity, background, novelty, contrast, depth/scope, domain, citations, links, relevant to other interests, rarity, subject matter, thought catalyst), full text (e.g., audience, novelty, type, possible content, utility), journal/publisher (e.g., novelty, main focus, perceived quality), and personal (e.g., competition, time requirements). Results further indicate that multiple criteria are used when making relevant, partially relevant, and not‐relevant judgments, and that most criteria can have either a positive or negative contribution to the relevance of a document. The criteria most frequently mentioned by study participants were content, followed by criteria characterizing the full text document. These findings may have implications for relevance feedback in information retrieval systems, suggesting that systems accept and utilize multiple positive and negative relevance criteria from users. Systems designers may want to focus on supporting content criteria followed by full text criteria as these may provide the greatest cost benefit.

Date: 2002
References: Add references at CitEc
Citations: View citations in EconPapers (1)

Downloads: (external link)
https://doi.org/10.1002/asi.10049

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:bla:jamist:v:53:y:2002:i:5:p:327-342

Ordering information: This journal article can be ordered from
https://doi.org/10.1002/(ISSN)1532-2890

Access Statistics for this article

More articles in Journal of the American Society for Information Science and Technology from Association for Information Science & Technology
Bibliographic data for series maintained by Wiley Content Delivery ().