VL-Meta: Vision-Language Models for Multimodal Meta-Learning

Han Ma, Baoyu Fan, Benjamin K. Ng () and Chan-Tong Lam
Additional contact information
Han Ma: Faculty of Applied Sciences, Macao Polytechnic University, Macao 999078, China
Baoyu Fan: Faculty of Applied Sciences, Macao Polytechnic University, Macao 999078, China
Benjamin K. Ng: Faculty of Applied Sciences, Macao Polytechnic University, Macao 999078, China
Chan-Tong Lam: Faculty of Applied Sciences, Macao Polytechnic University, Macao 999078, China

Mathematics, 2024, vol. 12, issue 2, 1-16

Abstract: Multimodal learning is a promising area in artificial intelligence (AI) that can make the model understand different kinds of data. Existing works are trying to re-train a new model based on pre-trained models that requires much data, computation power, and time. However, it is difficult to achieve in low-resource or small-sample situations. Therefore, we propose VL-Meta, Vision Language Models for Multimodal Meta Learning. It (1) presents the vision-language mapper and multimodal fusion mapper, which are light model structures, to use the existing pre-trained models to make models understand images to language feature space and save training data, computation power, and time; (2) constructs the meta-task pool that can only use a small amount of data to construct enough training data and improve the generalization of the model to learn the data knowledge and task knowledge; (3) proposes the token-level training that can align inputs with the outputs during training to improve the model performance; and (4) adopts the multi-task fusion loss to learn the different abilities for the models. It achieves a good performance on the Visual Question Answering (VQA) task, which shows the feasibility and effectiveness of the model. This solution can help blind or visually impaired individuals obtain visual information.

Keywords: vision-language models; multimodal learning; meta-learning; token-level training; visual question answering (search for similar items in EconPapers)
JEL-codes: C (search for similar items in EconPapers)
Date: 2024
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
https://www.mdpi.com/2227-7390/12/2/286/pdf (application/pdf)
https://www.mdpi.com/2227-7390/12/2/286/ (text/html)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:gam:jmathe:v:12:y:2024:i:2:p:286-:d:1319902

Access Statistics for this article

Mathematics is currently edited by Ms. Emma He

More articles in Mathematics from MDPI
Bibliographic data for series maintained by MDPI Indexing Manager ().