Prompt selection matters: enhancing text annotations for social sciences with large language models

Louis Abraham (), Charles Arnal () and Antoine Marie ()
Additional contact information
Louis Abraham: Université Paris 1 Panthéon-Sorbonne
Charles Arnal: Université Paris-Saclay
Antoine Marie: Institut Jean Nicod

Journal of Computational Social Science, 2025, vol. 8, issue 3, No 19, 20 pages

Abstract: Abstract Large Language Models have recently been applied to text annotation tasks from social sciences, equating or surpassing the performance of human workers at a fraction of the cost. However, very few inquiries in the social sciences have been made of the impact of prompt selection on labelling accuracy. In this study, we show that performance greatly varies between prompts, and we apply the method of automatic prompt optimization to systematically craft high quality prompts. We also provide the community with a simple, browser-based implementation of the method at https://prompt-ultra.github.io/ .

Keywords: Accuracy; Large language models; Text classification; Politics; Sentiment (search for similar items in EconPapers)
Date: 2025
References: Add references at CitEc
Citations:

Downloads: (external link)
http://link.springer.com/10.1007/s42001-025-00388-6 Abstract (text/html)
Access to the full text of the articles in this series is restricted.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:spr:jcsosc:v:8:y:2025:i:3:d:10.1007_s42001-025-00388-6

Ordering information: This journal article can be ordered from
http://www.springer. ... iences/journal/42001

DOI: 10.1007/s42001-025-00388-6

Access Statistics for this article

Journal of Computational Social Science is currently edited by Takashi Kamihigashi

More articles in Journal of Computational Social Science from Springer
Bibliographic data for series maintained by Sonal Shukla () and Springer Nature Abstracting and Indexing ().