Statistical Auditing of Non-transparent Expert Assessments

Michael Cain () and Stuart McLeay
Additional contact information
Michael Cain: Vietnam National University—HCMC

Sankhya B: The Indian Journal of Statistics, 2016, vol. 78, issue 2, No 7, 362-385

Abstract: Abstract A statistical model is developed for an audit of the assessments of a panel of experts when little information is made available beyond a final announcement of the individual assessed ratings given. The application is to the process for the research assessment exercise for UK universities. Based on the proportions of the publications a panel deems to be International standard, National standard or Unclassified, a department’s research output is rated by the panel on a seven point scale. The expert panel’s remit is carefully interpreted and the given ratings are modelled via an underlying trinomial random variable with a bivariate Normal approximation. A likelihood function is developed and maximised in order to obtain fitted ratings for all units of assessment. The model’s fitted values for the given ratings explain outcomes remarkably well and there are few mis-classifications; but there are some surprising outliers that do still require some explanation. The procedure illustrates well how Statisticians, surprisingly, might be able to model and audit for consistency the work of experts even if little or no information is provided, beyond vague prior published guidelines for the assessments and the final ratings given.

Keywords: Trinomial; Bivariate normal; Maximum likelihood estimation; MLE; Research assessment exercise; RAE; Primary 62P99, 65C60, 62-07; Secondary 62F30, 74P99 (search for similar items in EconPapers)
Date: 2016
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
http://link.springer.com/10.1007/s13571-016-0124-8 Abstract (text/html)
Access to the full text of the articles in this series is restricted.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:spr:sankhb:v:78:y:2016:i:2:d:10.1007_s13571-016-0124-8

Ordering information: This journal article can be ordered from
http://www.springer.com/statistics/journal/13571

DOI: 10.1007/s13571-016-0124-8

Access Statistics for this article

Sankhya B: The Indian Journal of Statistics is currently edited by Dipak Dey

More articles in Sankhya B: The Indian Journal of Statistics from Springer, Indian Statistical Institute
Bibliographic data for series maintained by Sonal Shukla () and Springer Nature Abstracting and Indexing ().