Computer Science Education in ChatGPT Era: Experiences from an Experiment in a Programming Course for Novice Programmers

Tomaž Kosar, Dragana Ostojić, Yu David Liu and Marjan Mernik ()
Additional contact information
Tomaž Kosar: Faculty of Electrical Engineering and Computer Science, University of Maribor, Koroška cesta 46, 2000 Maribor, Slovenia
Dragana Ostojić: Faculty of Electrical Engineering and Computer Science, University of Maribor, Koroška cesta 46, 2000 Maribor, Slovenia
Yu David Liu: Department of Computer Science, State University of New York at Binghamton (SUNY), 4400 Vestal Parkway East, Binghamton, NY 13902, USA
Marjan Mernik: Faculty of Electrical Engineering and Computer Science, University of Maribor, Koroška cesta 46, 2000 Maribor, Slovenia

Mathematics, 2024, vol. 12, issue 5, 1-22

Abstract: The use of large language models with chatbots like ChatGPT has become increasingly popular among students, especially in Computer Science education. However, significant debates exist in the education community on the role of ChatGPT in learning. Therefore, it is critical to understand the potential impact of ChatGPT on the learning, engagement, and overall success of students in classrooms. In this empirical study, we report on a controlled experiment with 182 participants in a first-year undergraduate course on object-oriented programming. Our differential study divided students into two groups, one using ChatGPT and the other not using it for practical programming assignments. The study results showed that the students’ performance is not influenced by ChatGPT usage (no statistical significance between groups with a p -value of 0.730), nor are the grading results of practical assignments ( p -value 0.760) and midterm exams ( p -value 0.856). Our findings from the controlled experiment suggest that it is safe for novice programmers to use ChatGPT if specific measures and adjustments are adopted in the education process.

Keywords: large language models; ChatGPT; artificial intelligence; controlled experiment; object-oriented programming; software engineering education (search for similar items in EconPapers)
JEL-codes: C (search for similar items in EconPapers)
Date: 2024
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
https://www.mdpi.com/2227-7390/12/5/629/pdf (application/pdf)
https://www.mdpi.com/2227-7390/12/5/629/ (text/html)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:gam:jmathe:v:12:y:2024:i:5:p:629-:d:1342623

Access Statistics for this article

Mathematics is currently edited by Ms. Emma He

More articles in Mathematics from MDPI
Bibliographic data for series maintained by MDPI Indexing Manager ().