Comparable Grading from Observational Data: Many-Facet Modelling with Soft Anchors

Håkon Otneim

No 2026/6, Discussion Papers from Norwegian School of Economics, Department of Business and Management Science

Abstract: This paper addresses grade comparability across exam cohorts when assessors and item sets change from year to year. Ordinal item scores reflect a mixture of student ability, item difficulty, and assessor severity; separating these components requires linking assumptions rarely verified empirically. We fit a sequence of Bayesian cumulative logit models to item-level scores from nine cohorts of an undergraduate statistics course. The setting is fully observational with no cross-grading and only partial assessor overlap, so cross-cohort alignment relies on repeated content used as anchors and on shared assessors. Sequential model expansion guided by posterior predictive checks reveals that treating anchors as having fixed difficulty across cohorts is inconsistent with the data. A soft-linking formulation, where linked items share a baseline difficulty but admit cohort-specific deviations regularised toward zero, removes the systematic misfit without discarding anchor information. Approximate cross-validation confirms that each modelling step improves out-of-sample predictive accuracy. Student ability estimates are robust to anchor specification (pairwise correlations exceeding 0.996), whereas cohort location estimates shift materially, which is a finding with direct consequences for grading policy. Using the recovered ability scale, we construct counterfactual grades and show that assessor severity is the dominant predictor of individual grade movement.

Keywords: comparable grading; many-facet measurement; soft anchors; cumulative logit model; Bayesian hierarchical models; PSIS-LOO cross-validation (search for similar items in EconPapers)
JEL-codes: C11 C25 C52 I21 I23 (search for similar items in EconPapers)
Pages: 32 pages
Date: 2026-05-22
References: Add references at CitEc
Citations:

Downloads: (external link)
https://hdl.handle.net/11250/5518775 Full text (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:hhs:nhhfms:2026_006

Access Statistics for this paper

More papers in Discussion Papers from Norwegian School of Economics, Department of Business and Management Science NHH, Department of Business and Management Science, Helleveien 30, N-5045 Bergen, Norway. Contact information at EDIRC.
Bibliographic data for series maintained by Stein Fossen ().