More human than human: measuring ChatGPT political bias

Fabio Motoki (), Valdemar Pinho Neto and Victor Rodrigues ()
Additional contact information
Fabio Motoki: University of East Anglia
Victor Rodrigues: Nova Educação

Public Choice, 2024, vol. 198, issue 1, No 2, 3-23

Abstract: Abstract We investigate the political bias of a large language model (LLM), ChatGPT, which has become popular for retrieving factual information and generating content. Although ChatGPT assures that it is impartial, the literature suggests that LLMs exhibit bias involving race, gender, religion, and political orientation. Political bias in LLMs can have adverse political and electoral consequences similar to bias from traditional and social media. Moreover, political bias can be harder to detect and eradicate than gender or racial bias. We propose a novel empirical design to infer whether ChatGPT has political biases by requesting it to impersonate someone from a given side of the political spectrum and comparing these answers with its default. We also propose dose-response, placebo, and profession-politics alignment robustness tests. To reduce concerns about the randomness of the generated text, we collect answers to the same questions 100 times, with question order randomized on each round. We find robust evidence that ChatGPT presents a significant and systematic political bias toward the Democrats in the US, Lula in Brazil, and the Labour Party in the UK. These results translate into real concerns that ChatGPT, and LLMs in general, can extend or even amplify the existing challenges involving political processes posed by the Internet and social media. Our findings have important implications for policymakers, media, politics, and academia stakeholders.

Keywords: Bias; Political bias; Large language models; ChatGPT (search for similar items in EconPapers)
JEL-codes: C10 C89 D83 L86 Z00 (search for similar items in EconPapers)
Date: 2024
References: View references in EconPapers View complete reference list from CitEc
Citations: View citations in EconPapers (3)

Downloads: (external link)
http://link.springer.com/10.1007/s11127-023-01097-2 Abstract (text/html)
Access to full text is restricted to subscribers.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:kap:pubcho:v:198:y:2024:i:1:d:10.1007_s11127-023-01097-2

Ordering information: This journal article can be ordered from
http://www.springer. ... ce/journal/11127/PS2

DOI: 10.1007/s11127-023-01097-2

Access Statistics for this article

Public Choice is currently edited by WIlliam F. Shughart II

More articles in Public Choice from Springer
Bibliographic data for series maintained by Sonal Shukla () and Springer Nature Abstracting and Indexing ().