Large Language Models in Student Assessment: Comparing ChatGPT and Human Graders

Magnus Lundgren

Papers from arXiv.org

Abstract: This study investigates the efficacy of large language models (LLMs) as tools for grading master-level student essays. Utilizing a sample of 60 essays in political science, the study compares the accuracy of grades suggested by the GPT-4 model with those awarded by university teachers. Results indicate that while GPT-4 aligns with human grading standards on mean scores, it exhibits a risk-averse grading pattern and its interrater reliability with human raters is low. Furthermore, modifications in the grading instructions (prompt engineering) do not significantly alter AI performance, suggesting that GPT-4 primarily assesses generic essay characteristics such as language quality rather than adapting to nuanced grading criteria. These findings contribute to the understanding of AI's potential and limitations in higher education, highlighting the need for further development to enhance its adaptability and sensitivity to specific educational assessment requirements.

Date: 2024-06
New Economics Papers: this item is included in nep-ain and nep-big
References: View complete reference list from CitEc
Citations:

Downloads: (external link)
http://arxiv.org/pdf/2406.16510 Latest version (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:arx:papers:2406.16510

Access Statistics for this paper

More papers in Papers from arXiv.org
Bibliographic data for series maintained by arXiv administrators ().