Systematic Evaluation of Nation-State Propaganda in LLM Outputs

Kevin T. Greene and Jacob N Shapiro

No 57buw_v1, SocArXiv from Center for Open Science

Abstract: The rapid growth and usability of AI is creating new risks to information quality, including the possibility that chatbot responses spread propaganda from state-backed influence campaigns. There is little systematic evidence regarding how frequently LLMs spread such narratives when responding to user queries. Previous attempts to assess these risks rely on small numbers of evaluation questions, use opaque query construction procedures, and fail to account for key sources of uncertainty in model responses. We address these gaps with an automated, reproducible pipeline that samples documented state-backed propaganda and news items, programmatically generates evaluation questions, and audits LLM-generated responses. We evaluate whether five prominent models: (1) reproduce arguments aligned with documented Kremlin propaganda narratives; (2) present the Kremlin perspective on events without providing corrective context; and (3) cite known propaganda outlets. We find substantially lower rates of propaganda-aligned responses and propaganda-source citation than prior small-sample work would suggest. These risks are particularly rare for questions resembling ordinary information searches about Ukraine. Our approach enables systematic comparisons across models and issue areas, supports ongoing monitoring, and could be extended to many other hotly-debated topics.

Date: 2026-05-20
References: Add references at CitEc
Citations:

Downloads: (external link)
https://osf.io/download/6a0b35cf683b7244a747972d/

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:osf:socarx:57buw_v1

DOI: 10.31219/osf.io/57buw_v1

Access Statistics for this paper

More papers in SocArXiv from Center for Open Science
Bibliographic data for series maintained by OSF ().