Evaluating GPT-4 Turbo`s Ability to Design English Reading Test Items for Language Learners

Sharifah Mofareh Alshehri and Mohammed S Alharbi

English Language Teaching, 2025, vol. 18, issue 7, 48

Abstract: This study evaluates the Generative Pre-trained Transformer (GPT-4 turbo) to design English reading multiple-choice questions (MCQs) for intermediate learners by addressing a deficiency in studies that examine GPT-4 turbo`s capabilities to generate MCQs using various prompt engineering techniques and evaluate their psychometric properties. Utilizing a descriptive quantitative method, a cohort of eight-item writers and 150 preparatory students participated in the study. Both the questionnaire and the generated online test were used to collect data. The findings reveal that zero-shot prompting demonstrates the highest level of agreement compared to few-shot prompting across three of the six aspects- text coherence, question stem quality, and answer options quality. The study finds that although all generated MCQs using few-shot prompting exhibit significantly higher discrimination values than those generated using zero-shot prompting, all MCQs displayed a low level of difficulty across all three prompt engineering techniques (zero-shot prompting, few-shot prompting (two-shot & four-shot). Taken together, this study proposes some profound implications for language assessment developers by illustrating how prompt design influences both the perceived and measured quality of AI-generated items. It also contributes to the literature by providing meaningful insights into using large language models, especially GPT-4 turbo, for AIG.

Date: 2025
References: Add references at CitEc
Citations:

Downloads: (external link)
https://ccsenet.org/journal/index.php/elt/article/download/0/0/51869/56435 (application/pdf)
https://ccsenet.org/journal/index.php/elt/article/view/0/51869 (text/html)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:ibn:eltjnl:v:18:y:2025:i:7:p:48

Access Statistics for this article

More articles in English Language Teaching from Canadian Center of Science and Education Contact information at EDIRC.
Bibliographic data for series maintained by Canadian Center of Science and Education ().