A Bayesian Approach to Ranking and Rater Evaluation

Jing Cao, S. Lynne Stokes and Song Zhang
Additional contact information
Song Zhang: University of Texas Southwestern Medical Center

Journal of Educational and Behavioral Statistics, 2010, vol. 35, issue 2, 194-214

Abstract: We develop a Bayesian hierarchical model for the analysis of ordinal data from multirater ranking studies. The model for a rater’s score includes four latent factors: one is a latent item trait determining the true order of items and the other three are the rater’s performance characteristics, including bias, discrimination, and measurement error in the ratings. The proposed approach aims at three goals. First, three Bayesian estimators are introduced to estimate the ranks of items. They all show a substantial improvement over the widely used score sums by using the information on the variable skill of the raters. Second, rater performance can be compared based on rater bias, discrimination, and measurement error. Third, a simulation-based decision-theoretic approach is described to determine the number of raters to employ. A simulation study and an analysis based on a grant review data set are presented.

Keywords: Bayesian hierarchical model; multirater ordinal data; ranking estimator; rater performance; number of raters (search for similar items in EconPapers)
Date: 2010
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
https://journals.sagepub.com/doi/10.3102/1076998609353116 (text/html)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:sae:jedbes:v:35:y:2010:i:2:p:194-214

DOI: 10.3102/1076998609353116

Access Statistics for this article

More articles in Journal of Educational and Behavioral Statistics
Bibliographic data for series maintained by SAGE Publications ().