Assessing the risks and opportunities posed by AI-enhanced influence operations on social media

Rolf Fredheim and James Pamment ()
Additional contact information
Rolf Fredheim: Lund University Psychological Defence Research Institute
James Pamment: Lund University Psychological Defence Research Institute

Place Branding and Public Diplomacy, 2025, vol. 21, issue 3, No 4, 319-326

Abstract: Abstract Large language models (LLMs) like GPT-4 have the potential to dramatically change the landscape of influence operations. They can generate persuasive, tailored content at scale, making campaigns using falsified content, such as disinformation and fake accounts, easier to produce. Advances in self-hosted open-source models have meant that adversaries can evade content moderation and security checks built into large commercial models such as those commercialised by Anthropic, Google, and OpenAI. New multi-lingual models make it easier than ever for foreign adversaries to pose as local actors. This article examines the heightened threats posed by synthetic media, as well as the potential that these tools hold for creating effective countermeasures. It begins with assessing the challenges posed by a toxic combination of automated bots, human-controlled troll accounts, and more targeted social engineering operations. However, the second part of the article assesses the potential for these same tools to improve detection. Promising countermeasures include running internal generative models to bolster training data for internal classifiers, detecting statistical anomalies, identifying output from common prompts, and building specialised classifiers optimised for specific monitoring needs.

Keywords: LLM; AI; Influence operations; Disinformation; Digital diplomacy (search for similar items in EconPapers)
Date: 2025
References: Add references at CitEc
Citations:

Downloads: (external link)
http://link.springer.com/10.1057/s41254-023-00322-5 Abstract (text/html)
Access to the full text of the articles in this series is restricted.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:pal:pbapdi:v:21:y:2025:i:3:d:10.1057_s41254-023-00322-5

Ordering information: This journal article can be ordered from
https://www.palgrave.com/gp/journal/41254

DOI: 10.1057/s41254-023-00322-5

Access Statistics for this article

Place Branding and Public Diplomacy is currently edited by Robert Govers and James Pamment

More articles in Place Branding and Public Diplomacy from Palgrave Macmillan
Bibliographic data for series maintained by Sonal Shukla () and Springer Nature Abstracting and Indexing ().