Item pool quality control in educational testing: change point model, compound risk, and sequential detection

Yunxiao Chen, Yi-Hsuan Lee and Xiaoou Li

LSE Research Online Documents on Economics from London School of Economics and Political Science, LSE Library

Abstract: In standardized educational testing, test items are reused in multiple test administrations. To ensure the validity of test scores, the psychometric properties of items should remain unchanged over time. In this article, we consider the sequential monitoring of test items, in particular, the detection of abrupt changes to their psychometric properties, where a change can be caused by, for example, leakage of the item or change of the corresponding curriculum. We propose a statistical framework for the detection of abrupt changes in individual items. This framework consists of (1) a multistream Bayesian change point model describing sequential changes in items, (2) a compound risk function quantifying the risk in sequential decisions, and (3) sequential decision rules that control the compound risk. Throughout the sequential decision process, the proposed decision rule balances the trade-off between two sources of errors, the false detection of prechange items, and the nondetection of postchange items. An item-specific monitoring statistic is proposed based on an item response theory model that eliminates the confounding from the examinee population which changes over time. Sequential decision rules and their theoretical properties are developed under two settings: the oracle setting where the Bayesian change point model is completely known and a more realistic setting where some parameters of the model are unknown. Simulation studies are conducted under settings that mimic real operational tests.

Keywords: standardized testing; test security; item preknowledge; sequential change point detection; multi-stream data; compound decision; multistream data (search for similar items in EconPapers)
JEL-codes: C1 (search for similar items in EconPapers)
Pages: 31 pages
Date: 2022-06-01
New Economics Papers: this item is included in nep-his and nep-ore
References: View references in EconPapers View complete reference list from CitEc
Citations:

Published in Journal of Educational and Behavioral Statistics, 1, June, 2022, 47(3), pp. 322 - 352. ISSN: 1076-9986

Downloads: (external link)
http://eprints.lse.ac.uk/112498/ Open access version. (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:ehl:lserod:112498

Access Statistics for this paper

More papers in LSE Research Online Documents on Economics from London School of Economics and Political Science, LSE Library LSE Library Portugal Street London, WC2A 2HD, U.K.. Contact information at EDIRC.
Bibliographic data for series maintained by LSERO Manager ().