Training Novices to Think, or Giving Them LLMs? Evidence from an RCT

Hemath Asirvatham, Chiara Betti, Rachel Brown, Arnaldo Camuffo, Aaron Chatterji, Chiara Fumagalli, Alfonso Gambardella, Myriam Mariani, Abhinav Pandey, Steve Ramos, Giovanni Salvucci and Vladimir Simic

No 21882, CEPR Discussion Papers from Centre for Economic Policy Research

Abstract: We study whether LLMs and cognitive skills influence the evaluation of novices’ intellectual tasks, when these evaluations depend on standard and well-defined performance criteria. This is a common case (junior employees, students’ exams) making our question relevant. In a 2×2 randomized control trial, we assign 1,053 first-year undergraduates in economics, management and finance to one of four conditions: a training on causal reasoning (as a relevant cognitive skill for intellectual tasks), ChatGPT Edu access (GPT-4o), both, or neither. We provide a real-world merchandising business problem and ask experts to evaluate the participants’ solutions according to standard metrics in marketing. We find that causal reasoning changes how students think, leading to mechanism-based solutions and a greater diversity of ideas. However, causal reasoning does not improve evaluations, unlike ChatGPT. Greater textual coherence and a higher number of ideas explain about half of the ChatGPT effect, suggesting that, for novices facing well-defined problems, LLMs produce solutions that resemble expert recommendations not only in textual quality, but also in content. We also find that, when combined with causal training, ChatGPT increases the use of coherent logic, falsification logic, and the attempt to understand why (mechanisms). Overall, we find that human causal reasoning produces effects independently of LLMs, particularly on the diversity of ideas, and therefore it is worth cultivating causal reasoning in a world in which students use LLMs. Probably less appreciated is that the value of diverse ideas also requires an evaluation system that demands diversity rather than standard solutions.

JEL-codes: J01 J08 M31 (search for similar items in EconPapers)
Date: 2026-08
References: Add references at CitEc
Citations:

Downloads: (external link)
https://cepr.org/publications/DP21882 (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:cpr:ceprdp:21882

Ordering information: This working paper can be ordered from
https://cepr.org/publications/DP21882

Access Statistics for this paper

More papers in CEPR Discussion Papers from Centre for Economic Policy Research 33 Great Sutton Street, London EC1V 0DX, UK.
Bibliographic data for series maintained by CEPR ().