Learning Modality Consistency and Difference Information with Multitask Learning for Multimodal Sentiment Analysis

Cheng Fang (), Feifei Liang, Tianchi Li and Fangheng Guan
Additional contact information
Cheng Fang: Key Laboratory of Civil Aviation Thermal Hazards Prevention and Emergency Response, Civil Aviation University of China, Tianjin 300300, China
Feifei Liang: China FAW (Nanjing) Technology Development Co., Ltd., Nanjing 211100, China
Tianchi Li: College of Electronic Information and Automation, Civil Aviation University of China, Tianjin 300300, China
Fangheng Guan: College of Electronic Information and Automation, Civil Aviation University of China, Tianjin 300300, China

Future Internet, 2024, vol. 16, issue 6, 1-17

Abstract: The primary challenge in Multimodal sentiment analysis (MSA) lies in developing robust joint representations that can effectively learn mutual information from diverse modalities. Previous research in this field tends to rely on feature concatenation to obtain joint representations. However, these approaches fail to fully exploit interactive patterns to ensure consistency and differentiation across different modalities. To address this limitation, we propose a novel framework for multimodal sentiment analysis, named CDML (Consistency and Difference using a Multitask Learning network). Specifically, CDML uses an attention mechanism to assign the attention weights of each modality efficiently. Adversarial training is used to obtain consistent information between modalities. Finally, the difference among the modalities is acquired by the multitask learning framework. Experiments on two benchmark MSA datasets, CMU-MOSI and CMU-MOSEI, showcase that our proposed method outperforms the seven existing approaches by at least 1.3% for Acc-2 and 1.7% for F1.

Keywords: multimodal sentiment analysis; attention mechanism; multitask learning; adversarial training (search for similar items in EconPapers)
JEL-codes: O3 (search for similar items in EconPapers)
Date: 2024
References: View complete reference list from CitEc
Citations:

Downloads: (external link)
https://www.mdpi.com/1999-5903/16/6/213/pdf (application/pdf)
https://www.mdpi.com/1999-5903/16/6/213/ (text/html)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:gam:jftint:v:16:y:2024:i:6:p:213-:d:1416436

Access Statistics for this article

Future Internet is currently edited by Ms. Grace You

More articles in Future Internet from MDPI
Bibliographic data for series maintained by MDPI Indexing Manager ().