AI scoring for international large-scale assessments using a deep learning model and multilingual data

Tomoya Okubo, Wayne Houlden, Paul Montuoro, Nate Reinertsen, Chi Sum Tse and Tanja Bastianic
Additional contact information
Tomoya Okubo: OECD
Wayne Houlden: Janison
Paul Montuoro: Janison
Nate Reinertsen: OECD
Chi Sum Tse: OECD
Tanja Bastianic: OECD

No 287, OECD Education Working Papers from OECD Publishing

Abstract: Artificial Intelligence (AI) scoring for constructed-response items, using recent advancements in multilingual, deep learning techniques utilising models pre-trained with a massive multilingual text corpus, is examined using international large-scale assessment data. Historical student responses to Reading and Science literacy cognitive items developed under the PISA analytical framework are used as training data for deep learning together with multilingual data to construct an AI model. The trained AI models are then used to score and the results compared with human-scored data. The score distributions estimated based on the AI-scored data and the human-scored data are highly consistent with each other; furthermore, even item-level psychometric properties of the majority of items showed high levels of agreement, although a few items showed discrepancies. This study demonstrates a practical procedure for using a multilingual data approach, and this new AI-scoring methodology reached a practical level of quality, even in the context of an international large-scale assessment.

Date: 2023-02-21
New Economics Papers: this item is included in nep-big
References: Add references at CitEc
Citations:

Downloads: (external link)
https://doi.org/10.1787/9918e1fb-en (text/html)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:oec:eduaab:287-en

Access Statistics for this paper

More papers in OECD Education Working Papers from OECD Publishing Contact information at EDIRC.
Bibliographic data for series maintained by ().