Comparative judgement without the fancy statistics

Ian Jones

No r487u_v1, OSF Preprints from Center for Open Science

Abstract: Comparative judgement methods for assessment are increasingly popular. They involve assessors making comparisons about the ‘quality’ of pairs of students’ work, and the comparisons are statistically modelled to produce scores. Recently Benton and Gallacher (2018, p.25) claimed that “much of the apparent advantage of [comparative judgement] can be explained by its use of fancy statistics”. They evidenced this by applying ‘fancy statistics’ to raw scores from multiple marked essays, and comparing the predictive value of raw scores with fancy statistics outcomes. Here I take the inverse approach and compare raw scores from comparative judgement assessments with fancy statistics outcomes. I reanalysed studies from peer-reviewed outlets where the prominent measure was based on comparative judgement. I report that raw scores reduced the reliability and validity of outcomes relative to fancy statistics in about one fifth of cases. I consider the implications of the findings for using comparative judgement in educational research.

Date: 2025-02-28
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
https://osf.io/download/67bd9d577210c9a6fe2e984d/

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:osf:osfxxx:r487u_v1

DOI: 10.31219/osf.io/r487u_v1

Access Statistics for this paper

More papers in OSF Preprints from Center for Open Science
Bibliographic data for series maintained by OSF ().