Certainty-Based Marking on Multiple-Choice Items: Psychometrics Meets Decision Theory

Qian Wu (), Monique Vanerum (), Anouk Agten (), Andrés Christiansen (), Frank Vandenabeele (), Jean-Michel Rigo () and Rianne Janssen ()
Additional contact information
Qian Wu: KU Leuven
Monique Vanerum: University of Hasselt
Anouk Agten: University of Hasselt
Andrés Christiansen: KU Leuven
Frank Vandenabeele: University of Hasselt
Jean-Michel Rigo: University of Hasselt
Rianne Janssen: KU Leuven

Psychometrika, 2021, vol. 86, issue 2, No 8, 518-543

Abstract: Abstract When a response to a multiple-choice item consists of selecting a single-best answer, it is not possible for examiners to differentiate between a response that is a product of knowledge and one that is largely a product of uncertainty. Certainty-based marking (CBM) is one testing format that requires examinees to express their degree of certainty on the response option they have selected, leading to an item score that depends both on the correctness of an answer and the certainty expressed. The expected score is maximized if examinees truthfully report their level of certainty. However, prospect theory states that people do not always make rational choices of the optimal outcome due to varying risk attitudes. By integrating a psychometric model and a decision-making perspective, the present study looks into the response behaviors of 334 first-year students of physiotherapy on six multiple-choice examinations with CBM in a case study. We used item response theory to model the objective probability of students giving a correct response to an item, and cumulative prospect theory to estimate their risk attitudes when students choose to report their certainty. The results showed that with the given CBM scoring matrix, students’ choices of a certainty level were affected by their risk attitudes. Students were generally risk averse and loss averse when they had a high success probability on an item, leading to an under-reporting of their certainty. Meanwhile, they were risk seeking in case of small success probabilities on the items, resulting in the over-reporting of certainty.

Keywords: certainty-based marking; multiple-choice questions; IRT; cumulative prospect theory; hierarchical Bayesian estimation (search for similar items in EconPapers)
Date: 2021
References: View references in EconPapers View complete reference list from CitEc
Citations: View citations in EconPapers (1)

Downloads: (external link)
http://link.springer.com/10.1007/s11336-021-09759-0 Abstract (text/html)
Access to the full text of the articles in this series is restricted.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:spr:psycho:v:86:y:2021:i:2:d:10.1007_s11336-021-09759-0

Ordering information: This journal article can be ordered from
http://www.springer. ... gy/journal/11336/PS2

DOI: 10.1007/s11336-021-09759-0

Access Statistics for this article

Psychometrika is currently edited by Irini Moustaki

More articles in Psychometrika from Springer, The Psychometric Society
Bibliographic data for series maintained by Sonal Shukla () and Springer Nature Abstracting and Indexing ().