This Candidate is [MASK]. Prompt-based Sentiment Extraction and Reference Letters

Fabian Slonimczyk

MPRA Paper from University Library of Munich, Germany

Abstract: I propose a relatively simple way to deploy pre-trained large language models (LLMs) in order to extract sentiment and other useful features from text data. The method, which I refer to as prompt-based sentiment extraction, offers multiple advantages over other methods used in economics and finance. In particular, it accepts the text input as is (without preprocessing) and produces a sentiment score that has a probability interpretation. Unlike other LLM-based approaches, it does not require any fine-tuning or labeled data. I apply my prompt-based strategy to a hand-collected corpus of confidential reference letters (RLs). I show that the sentiment contents of RLs are clearly reflected in job market outcomes. Candidates with higher average sentiment in their RLs perform markedly better regardless of the measure of success chosen. Moreover, I show that sentiment dispersion among letter writers negatively affects the job market candidate’s performance. I compare my sentiment extraction approach to other commonly used methods for sentiment analysis: ‘bag-of-words’ approaches, fine-tuned language models, and querying advanced chatbots. No other method can fully reproduce the results obtained by prompt-based sentiment extraction. Finally, I slightly modify the method to obtain ‘gendered’ sentiment scores (as in Eberhardt et al., 2023). I show that RLs written for female candidates emphasize ‘grindstone’ personality traits, whereas male candidates’ letters emphasize ‘standout’ traits. These gender differences negatively affect women’s job market outcomes.

Keywords: Large language models; text data; sentiment analysis; reference letters (search for similar items in EconPapers)
JEL-codes: C45 J16 M51 (search for similar items in EconPapers)
Date: 2025-10
New Economics Papers: this item is included in nep-ain
References: Add references at CitEc
Citations:

Downloads: (external link)
https://mpra.ub.uni-muenchen.de/126675/1/MPRA_paper_126675.pdf original version (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:pra:mprapa:126675

Access Statistics for this paper

More papers in MPRA Paper from University Library of Munich, Germany Ludwigstraße 33, D-80539 Munich, Germany. Contact information at EDIRC.
Bibliographic data for series maintained by Joachim Winter ().