Relevance Judgment Convergence Degree—A Measure of Assessors Inconsistency for Information Retrieval Datasets

Dengya Zhu (), Shastri L. Nimmagadda (), Kok Wai Wong () and Torsten Reiners ()
Additional contact information
Dengya Zhu: Curtin University
Shastri L. Nimmagadda: Curtin University
Kok Wai Wong: Murdoch University
Torsten Reiners: Curtin University

A chapter in Advances in Information Systems Development, 2023, pp 149-168 from Springer

Abstract: Abstract The quality of training/testing datasets is critical when a model is trained and evaluated by the annotated datasets. In Information Retrieval (IR), documents are annotated by human experts if they are relevant or not to a given query. Relevance judgment of human assessors is inherently subjective and dynamic. However, a small group of experts’ relevance judgment results are usually taken as ground truth to “objectively” evaluate the performance of an IR system. Recent trends intend to employ a group of judges, such as outsourcing, to alleviate the potentially biased judgment results stemmed from using only a single expert’s judgment. Nevertheless, different judges may have different opinions and may not agree with each other, and the inconsistency in human relevance judgment may affect the IR system evaluation results. Further, previous research focused mainly on the quality of documents, rather on the quality of queries submitted to an IR system. In this research, we introduce Relevance Judgment Convergence Degree (RJCD) to measure the quality of queries in the evaluation datasets. Experimental results reveal a strong correlation coefficient between the proposed RJCD score and the performance differences between two IR systems.

Keywords: Relevance judgment; Information retrieval; Information system development (search for similar items in EconPapers)
Date: 2023
References: Add references at CitEc
Citations:

There are no downloads for this item, see the EconPapers FAQ for hints about obtaining it.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:spr:lnichp:978-3-031-32418-5_9

Ordering information: This item can be ordered from
http://www.springer.com/9783031324185

DOI: 10.1007/978-3-031-32418-5_9

Access Statistics for this chapter

More chapters in Lecture Notes in Information Systems and Organization from Springer
Bibliographic data for series maintained by Sonal Shukla () and Springer Nature Abstracting and Indexing ().