Comparative analysis of language models in automated educational diagnostic tasks

María Teresa Nagua Velepucha
Additional contact information
María Teresa Nagua Velepucha: Faculty of Humanities and Social Sciences Technical University of Manabí

Applied Journal of Digital Innovation, 2026, vol. 2, 13

Abstract: This study presents a comparative analysis of language models for automated educational diagnosis, evaluating two specialized models (a feature-based ML model and a fine-tuned BERT) against three state-of-the-art Large Language Models (GPT-4, Claude 3, Gemini Pro). Using a mixed-methods approach on a corpus of 450 annotated student texts, the research assessed diagnostic accuracy, feedback quality, robustness, and hallucination rates. Results indicate that advanced LLMs, particularly GPT-4 and Claude 3 Opus, achieve superior diagnostic performance (F1-Score ≥ 0.83) and generate pedagogically higher-quality feedback, rated as more useful and actionable by expert judges. However, these models exhibit a critical weakness: a significant propensity for factual hallucinations (5.2% - 12.3%). In contrast, specialized models, while less sophisticated, offer near-perfect reliability and greater cost-efficiency. The study concludes that LLMs represent a qualitative leap in automated assessment but are not yet reliable for autonomous application. The most viable implementation model is AI-augmented education, where LLMs act as diagnostic assistants to teaching professionals, who provide essential oversight, validation, and personalization. This human-in-the-loop paradigm balances technological potential with pedagogical integrity and ethical responsibility.

Keywords: Automated assessment; large language models; educational diagnosis; formative feedback (search for similar items in EconPapers)
Date: 2026
References: View complete reference list from CitEc
Citations:

Downloads: (external link)
https://aj.reincisol.com/index.php/aj/article/view/13 Abstract page (text/html)
https://aj.reincisol.com/index.php/aj/article/download/13/29 Full text (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:ajd:ajdinn:v:2:y:2026:id:13

DOI: 10.65835/aj.2026.2.13

Access Statistics for this article

More articles in Applied Journal of Digital Innovation from Reincisol Publishing
Bibliographic data for series maintained by Homero Lastra ().