Using Multimodal Foundation Models for Detecting Fake Images on the Internet with Explanations

Vishnu S. Pendyala () and Ashwin Chintalapati
Additional contact information
Vishnu S. Pendyala: Department of Applied Data Science, San Jose State University, San Jose, CA 95192, USA
Ashwin Chintalapati: Department of Computer Science, Purdue University, West Lafayette, IN 47907, USA

Future Internet, 2024, vol. 16, issue 12, 1-16

Abstract: Generative AI and multimodal foundation models have fueled a proliferation of fake content on the Internet. This paper investigates if foundation models help detect and thereby contain the spread of fake images. The task of detecting fake images is a formidable challenge owing to its visual nature and intricate analysis. This paper details experiments using four multimodal foundation models, Llava, CLIP, Moondream2, and Gemini 1.5 Flash, to detect fake images. Explainable AI techniques such as Local Interpretable Model-Agnostic Explanations (LIME) and removal-based explanations are used to gain insights into the detection process. The dataset used comprised real images and fake images generated by a generative artificial intelligence tool called MidJourney. Results show that the models can achieve up to a 69% accuracy rate in detecting fake images in an intuitively explainable way, as confirmed by multiple techniques and metrics.

Keywords: misinformation containment; large multimodal models; explainable AI; image processing (search for similar items in EconPapers)
JEL-codes: O3 (search for similar items in EconPapers)
Date: 2024
References: View complete reference list from CitEc
Citations:

Downloads: (external link)
https://www.mdpi.com/1999-5903/16/12/432/pdf (application/pdf)
https://www.mdpi.com/1999-5903/16/12/432/ (text/html)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:gam:jftint:v:16:y:2024:i:12:p:432-:d:1525868

Access Statistics for this article

Future Internet is currently edited by Ms. Grace You

More articles in Future Internet from MDPI
Bibliographic data for series maintained by MDPI Indexing Manager ().