Hateful Memes Detection Based on Multi-Task Learning

Zhiyu Ma, Shaowen Yao, Liwen Wu, Song Gao and Yunqi Zhang ()
Additional contact information
Zhiyu Ma: Engineering Research Center of Cyberspace, Yunnan University, Kunming 650091, China
Shaowen Yao: Engineering Research Center of Cyberspace, Yunnan University, Kunming 650091, China
Liwen Wu: Engineering Research Center of Cyberspace, Yunnan University, Kunming 650091, China
Song Gao: Engineering Research Center of Cyberspace, Yunnan University, Kunming 650091, China
Yunqi Zhang: Engineering Research Center of Cyberspace, Yunnan University, Kunming 650091, China

Mathematics, 2022, vol. 10, issue 23, 1-16

Abstract: With the popularity of posting memes on social platforms, the severe negative impact of hateful memes is growing. As existing detection models have lower detection accuracy than humans, hateful memes detection is still a challenge to statistical learning and artificial intelligence. This paper proposed a multi-task learning method consisting of a primary multimodal task and two unimodal auxiliary tasks to address this issue. We introduced a self-supervised generation strategy in auxiliary tasks to generate unimodal auxiliary labels automatically. Meanwhile, we used BERT and RESNET as the backbone for text and image classification, respectively, and then fusion them with a late fusion method. In the training phase, the backward guidance technique and the adaptive weight adjustment strategy were used to capture the consistency and variability between different modalities, numerically improving the hateful memes detection accuracy and the generalization and robustness of the model. The experiment conducted on the Facebook AI multimodal hateful memes dataset shows that the prediction accuracy of our model outperformed the comparing models.

Keywords: hateful memes; deep learning; multimodal data; multi-task learning; self-supervised (search for similar items in EconPapers)
JEL-codes: C (search for similar items in EconPapers)
Date: 2022
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
https://www.mdpi.com/2227-7390/10/23/4525/pdf (application/pdf)
https://www.mdpi.com/2227-7390/10/23/4525/ (text/html)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:gam:jmathe:v:10:y:2022:i:23:p:4525-:d:989103

Access Statistics for this article

Mathematics is currently edited by Ms. Emma He

More articles in Mathematics from MDPI
Bibliographic data for series maintained by MDPI Indexing Manager ().