Enhancing representation in radiography-reports foundation model: a granular alignment algorithm using masked contrastive learning

Weijian Huang, Cheng Li, Hong-Yu Zhou, Hao Yang, Jiarun Liu, Yong Liang, Hairong Zheng, Shaoting Zhang and Shanshan Wang ()
Additional contact information
Weijian Huang: Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences
Cheng Li: Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences
Hong-Yu Zhou: Harvard Medical University
Hao Yang: Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences
Jiarun Liu: Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences
Yong Liang: Pengcheng Laboratory
Hairong Zheng: Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences
Shaoting Zhang: Qingyuan Research Institute, Shanghai Jiao Tong University
Shanshan Wang: Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences

Nature Communications, 2024, vol. 15, issue 1, 1-12

Abstract: Abstract Recently, multi-modal vision-language foundation models have gained significant attention in the medical field. While these models offer great opportunities, they still face crucial challenges, such as the requirement for fine-grained knowledge understanding in computer-aided diagnosis and the capability of utilizing very limited or even no task-specific labeled data in real-world clinical applications. In this study, we present MaCo, a masked contrastive chest X-ray foundation model that tackles these challenges. MaCo explores masked contrastive learning to simultaneously achieve fine-grained image understanding and zero-shot learning for a variety of medical imaging tasks. It designs a correlation weighting mechanism to adjust the correlation between masked chest X-ray image patches and their corresponding reports, thereby enhancing the model’s representation learning capabilities. To evaluate the performance of MaCo, we conducted extensive experiments using 6 well-known open-source X-ray datasets. The experimental results demonstrate the superiority of MaCo over 10 state-of-the-art approaches across tasks such as classification, segmentation, detection, and phrase grounding. These findings highlight the significant potential of MaCo in advancing a wide range of medical image analysis tasks.

Date: 2024
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
https://www.nature.com/articles/s41467-024-51749-0 Abstract (text/html)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:nat:natcom:v:15:y:2024:i:1:d:10.1038_s41467-024-51749-0

Ordering information: This journal article can be ordered from
https://www.nature.com/ncomms/

DOI: 10.1038/s41467-024-51749-0

Access Statistics for this article

Nature Communications is currently edited by Nathalie Le Bot, Enda Bergin and Fiona Gillespie

More articles in Nature Communications from Nature
Bibliographic data for series maintained by Sonal Shukla () and Springer Nature Abstracting and Indexing ().