Mixture polarization in inter-rater agreement analysis: a Bayesian nonparametric index

Giuseppe Mignemi (), Antonio Calcagnì (), Andrea Spoto () and Ioanna Manolopoulou ()
Additional contact information
Giuseppe Mignemi: University of Padova
Antonio Calcagnì: DPSS, University of Padova
Andrea Spoto: University of Padova
Ioanna Manolopoulou: University College London

Statistical Methods & Applications, 2024, vol. 33, issue 1, No 13, 325-355

Abstract: Abstract In several observational contexts where different raters evaluate a set of items, it is common to assume that all raters draw their scores from the same underlying distribution. However, a plenty of scientific works have evidenced the relevance of individual variability in different type of rating tasks. To address this issue the intra-class correlation coefficient (ICC) has been used as a measure of variability among raters within the Hierarchical Linear Models approach. A common distributional assumption in this setting is to specify hierarchical effects as independent and identically distributed from a normal with the mean parameter fixed to zero and unknown variance. The present work aims to overcome this strong assumption in the inter-rater agreement estimation by placing a Dirichlet Process Mixture over the hierarchical effects’ prior distribution. A new nonparametric index $$\lambda$$ λ is proposed to quantify raters polarization in presence of group heterogeneity. The model is applied on a set of simulated experiments and real world data. Possible future directions are discussed.

Keywords: Bayesian nonparametrics; Inter-rater agreement; Dirichlet process mixture; Hierarchical Bayesian models (search for similar items in EconPapers)
Date: 2024
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
http://link.springer.com/10.1007/s10260-023-00741-x Abstract (text/html)
Access to the full text of the articles in this series is restricted.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:spr:stmapp:v:33:y:2024:i:1:d:10.1007_s10260-023-00741-x

Ordering information: This journal article can be ordered from
http://www.springer. ... cs/journal/10260/PS2

DOI: 10.1007/s10260-023-00741-x

Access Statistics for this article

Statistical Methods & Applications is currently edited by Tommaso Proietti

More articles in Statistical Methods & Applications from Springer, Società Italiana di Statistica
Bibliographic data for series maintained by Sonal Shukla () and Springer Nature Abstracting and Indexing ().