Assessing ChatGPT's ability to detect and correct programming errors in stata do-files

Ricardo Mora

UC3M Working papers. Economics from Universidad Carlos III de Madrid. Departamento de Economía

Abstract: This paper evaluates the efectiveness of ChatGPT in debugging Stata, a proprietary econometric software, with a focus on accuracy, hallucination tendencies, and the presence of illusory expertise. By systematically testing ChatGPT-3.5-turbo and ChatGPT-4o across diferent error types and user expertise levels, the study finds that while ChatGPT signifcantly improves debugging performance in open-book mode, its accuracy remainshighly dependent on error complexity. Hallucinations, where the model generates plausible but incorrect error explanations, are particularly frequent in syntax errors and persist even with access to error logs. Advanced scripts present the greatest challenge, with ChatGPT- 4o achieving only a 35.7 percent success rate and exhibiting a 25 percent hallucination rate despite access to error messages. The findings align with broader research on largelanguage models in coding assistance, demonstrating that ChatGPT struggles with complex debugging tasks in proprietary software environments like Stata, where integration is restricted.

Keywords: LLMs; Programming; Econometrics; software; Proprietary; software (search for similar items in EconPapers)
Date: 2025-02-13
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
https://e-archivo.uc3m.es/rest/api/core/bitstreams ... 91c3f0839c68/content (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:cte:werepe:45949

Access Statistics for this paper

More papers in UC3M Working papers. Economics from Universidad Carlos III de Madrid. Departamento de Economía
Bibliographic data for series maintained by Ana Poveda ().