Uncovering the semantics of concepts using GPT-4 and Other recent large language models

Gaël Le Mens (gael.le-mens@upf.edu), Balász Kovács, Michael T. Hannan and Guillem Pros
Additional contact information
Gaël Le Mens: https://www.upf.edu/web/econ/faculty/-/asset_publisher/6aWmmXf28uXT/persona/id/3420330

Economics Working Papers from Department of Economics and Business, Universitat Pompeu Fabra

Abstract: Recently, the world's attention has been captivated by Large Language Models (LLMs) thanks to OpenAI's Chat-GPT, which rapidly proliferated as an app powered by GPT-3 and now its successor, GPT-4. If these LLMs produce human-like text, the semantic spaces they construct likely align with those used by humans for interpreting and generating language. This suggests that social scientists could use these LLMs to construct measures of semantic similarity that match human judgment. In this article, we provide an empirical test of this intuition. We use GPT-4 to construct a new measure of typicality– the similarity of a text document to a concept or category. We evaluate its performance against other model-based typicality measures in terms of their correspondence with human typicality ratings. We conduct this comparative analysis in two domains: the typicality of books in literary genres (using an existing dataset of book descriptions) and the typicality of tweets authored by US Congress members in the Democratic and Republican parties (using a novel dataset). The GPT-4 Typicality measure not only meets or exceeds the current state-of-the-art but accomplishes this without any model training. This is a breakthrough because the previous state-of-the-art measure required fine-tuning a model (a BERT text classifier) on hundreds of thousands of text documents to achieve its performance. Our comparative analysis emphasizes the need for systematic empirical validation of measures based on LLMs: several measures based on other recent LLMs achieve at best a moderate correspondence with human judgments.

Keywords: categories; concepts; deep learning; typicality; GPT; chatGPT; BERT; Similarity (search for similar items in EconPapers)
JEL-codes: C18 C52 (search for similar items in EconPapers)
Date: 2023-06
New Economics Papers: this item is included in nep-ain and nep-cmp
References: View references in EconPapers View complete reference list from CitEc
Citations: View citations in EconPapers (1)

Downloads: (external link)
https://econ-papers.upf.edu/papers/1864.pdf Whole Paper (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:upf:upfgen:1864

Access Statistics for this paper

More papers in Economics Working Papers from Department of Economics and Business, Universitat Pompeu Fabra
Bibliographic data for series maintained by (william.carlson@upf.edu this e-mail address is bad, please contact repec@repec.org).