Structured AI Socratic Homework in Higher Education: Implementation, Student Experience, and Evidence from a Pre-Registered Field Study

Margarita Zabelina
Additional contact information
Margarita Zabelina: University of Pittsburgh

No c3ysd_v1, EdArXiv from Center for Open Science

Abstract: Background: Structured AI Socratic tutoring — in which a large language model guides students through multi-step problems using iterative questioning rather than direct answers — offers a scalable, zero-cost approach to formative homework in higher education. Existing literature on AI in education relies heavily on unstructured ChatGPT use and focuses on language disciplines, leaving structured Socratic designs in quantitative courses largely unexplored. Rigorous empirical evidence on design, implementation, and academic effectiveness remains scarce. Objectives: This pre-registered pilot study documents the design and iterative implementation of an AI Socratic homework intervention and evaluates its academic and experiential outcomes compared to a conventional homework platform. Methods: A quasi-experimental design with historical controls compared a Spring 2026 cohort using AI Socratic homework (n = 76) against a Fall 2025 cohort using a publisher platform (n = 84) in an undergraduate quantitative course, with identical exams, instructor, grader, and course materials across semesters. Mixed methods included Welch t-tests and equivalence testing for academic outcomes, pre-post surveys for student experience and AI literacy, and thematic analysis of open-ended responses. Results and Conclusions: Academic outcomes were statistically equivalent with a small positive effect favoring the AI group (d = 0.26), meeting a pre-registered practical significance threshold. Students preferred AI homework independent of cost (52%, p = .0002) and reported substantial AI comfort gains (71%). Eighty-nine percent used AI beyond required assignments, indicating spillover AI literacy development. The study's central contribution is a documented twelve-cycle prompt design and iteration framework constituting transferable implementation knowledge for instructors in any quantitative discipline.

Date: 2026-07-03
References: Add references at CitEc
Citations:

Downloads: (external link)
https://osf.io/download/6a43eb96328efa3374932f45/

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:osf:edarxi:c3ysd_v1

DOI: 10.31219/osf.io/c3ysd_v1

Access Statistics for this paper

More papers in EdArXiv from Center for Open Science
Bibliographic data for series maintained by OSF ().