A comment on "An Empirical Investigation of the Impact of ChatGPT on Creativity"

Jakub Prochazka, Jing Zhou, Ioana-Florina Coita and Shumi Akhtar

No 274, I4R Discussion Paper Series from The Institute for Replication (I4R)

Abstract: This report describes a computational reproduction of Lee and Chung's (2024) paper, which examined whether using ChatGPT (GPT‑3.5) enhances creativity in adults compared to web-search assistance or no assistance. The authors presented six randomized controlled experiments showing that ChatGPT-assisted responses were assessed as significantly more creative (effect sizes ranging from Cohen's d = 0.32 to 1.88). These effects were robust across diverse tasks and contexts. We first computationally reproduced all the main results using the original dataset and code, obtaining the same results as those presented by the authors in their paper. During the reproduction process, we identified two minor coding errors and one typographical error in the original table, none of which affected the substantive conclusions. Second, we performed a recreate reproduction for the main analysis in Experiments 1 and 3 by writing new R code. Our results again matched the results presented in the original paper. Overall, based on our analyses, the study is fully computationally reproducible from raw data, although only with access to the original code, due to undocumented cleaning steps, some non-described exclusion criteria, and missing codebooks. Several analyses in the original paper showed that ideas generated by ChatGPT are rated as similarly creative regardless of whether people modify them or not. We contributed to this conclusion by introducing a new robustness check using response time as a proxy for human effort in modifying ChatGPT outputs. Using data from Experiment 3, we found no significant correlation between response time and creativity in the ChatGPT condition (r = −.079, p = .449) and no moderating effect of response time on the influence of using ChatGPT on creativity. This suggests that human effort does not incrementally improve creativity beyond ChatGPT's contribution. Taken together, our findings support the original claim that using ChatGPT increases creativity regardless of the human input.

Date: 2025
References: Add references at CitEc
Citations:

Downloads: (external link)
https://www.econstor.eu/bitstream/10419/333869/1/I4R-DP274.pdf (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:zbw:i4rdps:274

Access Statistics for this paper

More papers in I4R Discussion Paper Series from The Institute for Replication (I4R)
Bibliographic data for series maintained by ZBW - Leibniz Information Centre for Economics ().