Evaluating the role of ChatGPT in enhancing EFL writing assessments in classroom settings: A preliminary investigation

Junfei Li, Jinyan Huang (), Wenyan Wu and Patrick B. Whipple
Additional contact information
Junfei Li: Shanghai Dianji University
Jinyan Huang: Jiangsu University
Wenyan Wu: Beijing Institute of Graphic Communication
Patrick B. Whipple: The Genesee Valley Board of Cooperative Educational Services

Palgrave Communications, 2024, vol. 11, issue 1, 1-9

Abstract: Abstract Using generalizability (G-) theory and qualitative feedback analysis, this study evaluated the role of ChatGPT in enhancing English-as-a-foreign-language (EFL) writing assessments in classroom settings. The primary objectives were to assess the reliability of the holistic scores assigned to EFL essays by ChatGPT versions 3.5 and 4 compared to college English teachers and to evaluate the relevance of the qualitative feedback provided by these versions of ChatGPT. The study analyzed 30 College English Test Band 4 (CET-4) essays written by non-English majors at a university in Beijing, China. ChatGPT versions 3.5 and 4, along with four college English teachers, served as raters. They scored the essays holistically following the CET-4 scoring rubric and also provided qualitative feedback on the language, content, and organization of these essays. The G-theory analysis revealed that the scoring reliability of ChatGPT3.5 was consistently lower than that of the teacher raters; however, ChatGPT4 demonstrated consistently higher reliability coefficients than the teachers. The qualitative feedback analysis indicated that both ChatGPT3.5 and 4 consistently provided more relevant feedback on the EFL essays than the teacher raters. Furthermore, ChatGPT versions 3.5 and 4 were equally relevant across the language, content, and organization aspects of the essays, whereas the teacher raters generally focused more on language but provided less relevant feedback on content and organization. Consequently, ChatGPT versions 3.5 and 4 could be useful AI tools for enhancing EFL writing assessments in classroom settings. The implications of adopting ChatGPT for classroom writing assessments are discussed.

Date: 2024
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
http://link.springer.com/10.1057/s41599-024-03755-2 Abstract (text/html)
Access to full text is restricted to subscribers.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:pal:palcom:v:11:y:2024:i:1:d:10.1057_s41599-024-03755-2

Ordering information: This journal article can be ordered from
https://www.nature.com/palcomms/about

DOI: 10.1057/s41599-024-03755-2

Access Statistics for this article

More articles in Palgrave Communications from Palgrave Macmillan
Bibliographic data for series maintained by Sonal Shukla () and Springer Nature Abstracting and Indexing ().