Chance‐Corrected Interrater Agreement Statistics for Two‐Rater Dichotomous Responses: A Method Review With Comparative Assessment Under Possibly Correlated Decisions

Zizhong Tian, Vernon M. Chinchilli, Chan Shen and Shouhao Zhou

International Statistical Review, 2025, vol. 93, issue 2, 199-221

Abstract: Measurement of the interrater agreement (IRA) is critical for assessing the reliability and validity of ratings in various disciplines. While numerous IRA statistics have been developed, there is a lack of guidance on selecting appropriate measures especially when raters' decisions could be correlated. To address this gap, we review a family of chance‐corrected IRA statistics for two‐rater dichotomous‐response cases, a fundamental setting that not only serves as the theoretical foundation for categorical‐response or multirater IRA methods but is also practically dominant in most empirical studies, and we propose a novel data‐generating framework to simulate correlated decision processes between raters. Subsequently, a new estimand, which calibrates the ‘true’ chance‐corrected IRA, is introduced while accounting for the potential ‘probabilistic certainty’. Extensive simulations were conducted to evaluate the performance of the reviewed IRA methods under various practical scenarios and were summarised by an agglomerative hierarchical clustering analysis. Finally, we provide recommendations for selecting appropriate IRA statistics based on outcome prevalence and rater characteristics and highlight the need for further advancements in IRA estimation methodologies.

Date: 2025
References: Add references at CitEc
Citations:

Downloads: (external link)
https://doi.org/10.1111/insr.12606

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:bla:istatr:v:93:y:2025:i:2:p:199-221

Ordering information: This journal article can be ordered from
http://www.blackwell ... bs.asp?ref=0306-7734

Access Statistics for this article

International Statistical Review is currently edited by Eugene Seneta and Kees Zeelenberg

More articles in International Statistical Review from International Statistical Institute Contact information at EDIRC.
Bibliographic data for series maintained by Wiley Content Delivery ().