Using Large Language Models for Text Classification in Experimental Economics

Can Celebi and Stefan Penczynski
Additional contact information
Can Celebi: University of Mannheim
Stefan Penczynski: School of Economics and Centre for Behavioural and Experimental Social Science, University of East Anglia

No 24-01, Working Paper series, University of East Anglia, Centre for Behavioural and Experimental Social Science (CBESS) from School of Economics, University of East Anglia, Norwich, UK.

Abstract: In our study, we compare the classification capabilities of GPT-3.5 and GPT-4 with human annotators using text data from economic experiments. We analysed four text corpora, focusing on two domains: promises and strategic reasoning. Starting with prompts close to those given to human annotators, we subsequently explored alternative prompts to investigate the effect of varying classification instructions and degrees of background information on the models' classification performance. Additionally, we varied the number of examples in a prompt (few-shot vs zero-shot) and the use of the zero-shot "Chain of Thought" prompting technique. Our findings show that GPT-4's performance is comparable to human annotators, achieving accuracy levels near or over 90% in three tasks, and in the most challenging task of classifying strategic thinking in asymmetric coordination games, it reaches an accuracy level above 70%.

Keywords: Text Classification; GPT; Strategic Thinking; Promises (search for similar items in EconPapers)
Date: 2024-06
New Economics Papers: this item is included in nep-ain, nep-big, nep-cmp and nep-exp
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
https://ueaeco.github.io/working-papers/papers/cbess/UEA-CBESS-24-01.pdf main text (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:uea:wcbess:24-01

Ordering information: This working paper can be ordered from
Reception, School of Economics, University of East Anglia, Norwich Research Park, Norwich, NR4 7TJ, UK

Access Statistics for this paper

More papers in Working Paper series, University of East Anglia, Centre for Behavioural and Experimental Social Science (CBESS) from School of Economics, University of East Anglia, Norwich, UK. Contact information at EDIRC.
Bibliographic data for series maintained by Cara Liggins ().