Scoring and Keying Multiple Choice Tests: A Case Study in Irrationality

Maya Bar-Hillel (), David Budescu () and Yigal Attali ()

Discussion Paper Series from The Federmann Center for the Study of Rationality, the Hebrew University, Jerusalem

Abstract: We offer a case-study in irrationality, showing that even in a high stakes, deliberate context, highly intelligent professionals may adopt dominated practices. Multiple-choice tests (MCTs) enjoy many advantages that made them popular tools in educational and psychological measurement. But they suffer from the so-called guessing problem: test-makers cannot distinguish lucky guesses from answers based on knowledge. One way professional test-makers have dealt with this problem is by attempting to lower the incentive to guess, though penalizing errors (called formula scoring). Another is to rid tests of various cues (e.g., a preponderance of correct answers in middle positions) that might help testwise test-takers guess at better than chance odds. Key balancing is the strategy test-takers adopted for ridding tests of positional biases. We show that formula scoring and key balancing, though widespread and intuitively appealing, are in fact "irrational" practices. They do not dispose of the guessing problem and are fraught with problems of their own. Yet they persist, even in the presence of more rational alternatives: Number right scoring is superior to formula scoring, and key randomization is superior to key balancing.

Keywords: formula scoring; guessing; key balancing; multiple-choice tests; randomization; rationality; testwiseness (search for similar items in EconPapers)
Pages: 10 pages
Date: 2004-10
References: View complete reference list from CitEc
Citations:

Published in Mind & Society, 2005, vol. 4, pp. 3–12.

Downloads: (external link)
http://ratio.huji.ac.il/sites/default/files/publications/dp370.pdf (application/pdf)
Our link check indicates that this URL is bad, the error code is: 404 Not Found (http://ratio.huji.ac.il/sites/default/files/publications/dp370.pdf [302 Moved Temporarily]--> https://ratio.huji.ac.il/sites/default/files/publications/dp370.pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:huj:dispap:dp370

Access Statistics for this paper

More papers in Discussion Paper Series from The Federmann Center for the Study of Rationality, the Hebrew University, Jerusalem Contact information at EDIRC.
Bibliographic data for series maintained by Michael Simkin ().