Noise Improves Multimodal Machine Translation: Rethinking the Role of Visual Context

Xinyu Ma, Jun Rao and Xuebo Liu ()
Additional contact information
Xinyu Ma: School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen 518055, China
Jun Rao: School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen 518055, China
Xuebo Liu: School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen 518055, China

Mathematics, 2025, vol. 13, issue 11, 1-16

Abstract: Multimodal Machine Translation (MMT) has long been assumed to outperform traditional text-only MT by leveraging visual information. However, recent studies challenge this assumption, showing that MMT models perform similarly even when tested without images or with mismatched images. This raises fundamental questions about the actual utility of visual information in MMT, which this work aims to investigate. We first revisit commonly used image-must and image-free MMT approaches, identifying that suboptimal performance may stem from insufficiently robust baseline models. To further examine the role of visual information, we propose a novel visual type regularization method and introduce two probing tasks—Visual Contribution Probing and Modality Relationship Probing—to analyze whether and how visual features influence a strong MMT model. Surprisingly, our findings on a mainstream dataset indicate that the gains from visual information are marginal. We attribute this improvement primarily to a regularization effect, which can be replicated using random noise. Our results suggest that the MMT community should critically re-evaluate baseline models, evaluation metrics, and dataset design to advance multimodal learning meaningfully.

Keywords: machine translation; multimodal; probing tasks (search for similar items in EconPapers)
JEL-codes: C (search for similar items in EconPapers)
Date: 2025
References: Add references at CitEc
Citations:

Downloads: (external link)
https://www.mdpi.com/2227-7390/13/11/1874/pdf (application/pdf)
https://www.mdpi.com/2227-7390/13/11/1874/ (text/html)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:gam:jmathe:v:13:y:2025:i:11:p:1874-:d:1671305

Access Statistics for this article

Mathematics is currently edited by Ms. Emma He

More articles in Mathematics from MDPI
Bibliographic data for series maintained by MDPI Indexing Manager ().