Designing More Informative Tests: Separating Execution from Recognition

Andrew Caplin and Leo Zhu

No 35232, NBER Working Papers from National Bureau of Economic Research, Inc

Abstract: Tests are widely used to measure ability, yet performance on a test often reflects more than the ability to execute assigned tasks. It also reflects the ability to recognize which tasks are worth attempting, how they should be prioritized, and how effort should be allocated under uncertainty. This paper studies how tests can be designed to separate these capabilities. We model a test as a sequential decision problem. Tasks differ in difficulty, their ordering is uncertain, and examinees may acquire costly information about that ordering before choosing how to proceed. The testing environment is the informational structure surrounding the realized test: in particular, the examinee's beliefs about how task difficulty has been arranged. Performance is therefore generated by an optimal recognition–execution policy, not by execution skill alone. The analysis delivers two negative results. First, even in the simplest two-task environment, a single score exhibits dimensional collapse: distinct combinations of execution skill and recognition capability generate identical expected scores. Second, with three tasks, the relationship between capabilities and scores becomes environment-dependent: changing beliefs about task ordering can change which actions are considered and how capabilities translate into performance. These results imply that standard scores are not generally informative enough to separate the capabilities that generate performance. This matters because scores are used to summarize what individuals can do and to guide downstream decisions about placement, training, and instruction. If a test does not separately reveal execution and recognition, it provides limited guidance about which capability is strong, which is weak, and where improvement should be directed. We then show how more informative tests can be designed. Under a simple communicability constraint, two canonical environments—ordered and randomized tests—induce distinct relationships between capabilities and scores. In an ordered test, recognition is suppressed and performance isolates execution. In a randomized test, recognition is activated and performance reflects both execution and recognition. Observing performance across these environments separates capabilities that are confounded in any single score. The paper reframes testing as a problem of informational design: tests should be designed not only to record performance, but to reveal the distinct capabilities that generate it.

JEL-codes: C90 D83 I21 I24 J24 O33 (search for similar items in EconPapers)
Date: 2026-05
Note: ED
References: Add references at CitEc
Citations:

Downloads: (external link)
http://www.nber.org/papers/w35232.pdf (application/pdf)
Access to the full text is generally limited to series subscribers, however if the top level domain of the client browser is in a developing country or transition economy free access is provided. More information about subscriptions and free access is available at http://www.nber.org/wwphelp.html. Free access is also available to older working papers.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:nbr:nberwo:35232

Ordering information: This working paper can be ordered from
http://www.nber.org/papers/w35232
The price is Paper copy available by mail.

Access Statistics for this paper

More papers in NBER Working Papers from National Bureau of Economic Research, Inc National Bureau of Economic Research, 1050 Massachusetts Avenue Cambridge, MA 02138, U.S.A.. Contact information at EDIRC.
Bibliographic data for series maintained by ().