An explainable longitudinal multi-modal fusion model for predicting neoadjuvant therapy response in women with breast cancer

Yuan Gao, Sofia Ventura-Diaz, Xin Wang, Muzhen He, Zeyan Xu, Arlene Weir, Hong-Yu Zhou, Tianyu Zhang, Frederieke H. Duijnhoven, Luyi Han, Xiaomei Li, Anna D’Angelo, Valentina Longo, Zaiyi Liu, Jonas Teuwen, Marleen Kok, Regina Beets-Tan, Hugo M. Horlings, Tao Tan () and Ritse Mann
Additional contact information
Yuan Gao: Maastricht University Medical Centre
Sofia Ventura-Diaz: St Joseph’s Healthcare Hamilton
Xin Wang: Maastricht University Medical Centre
Muzhen He: Shengli Clinical Medical College of Fujian Medical University, Fuzhou University Affiliated Provincial Hospital
Zeyan Xu: The Third Affiliated Hospital of Kunming Medical University
Arlene Weir: Cork University Hospital
Hong-Yu Zhou: Harvard Medical School
Tianyu Zhang: Maastricht University Medical Centre
Frederieke H. Duijnhoven: Netherlands Cancer Institute
Luyi Han: Netherlands Cancer Institute
Xiaomei Li: The Second Clinical Medical College of Jinan University
Anna D’Angelo: Oncological Radiotherapy and Hematology, Fondazione Policlinico Universitario ‘A. Gemelli’ IRCCS
Valentina Longo: Oncological Radiotherapy and Hematology, Fondazione Policlinico Universitario ‘A. Gemelli’ IRCCS
Zaiyi Liu: Southern Medical University
Jonas Teuwen: Netherlands Cancer Institute
Marleen Kok: Netherlands Cancer Institute
Regina Beets-Tan: Maastricht University Medical Centre
Hugo M. Horlings: Netherlands Cancer Institute
Tao Tan: Macao Polytechnic University
Ritse Mann: Netherlands Cancer Institute

Nature Communications, 2024, vol. 15, issue 1, 1-17

Abstract: Abstract Multi-modal image analysis using deep learning (DL) lays the foundation for neoadjuvant treatment (NAT) response monitoring. However, existing methods prioritize extracting multi-modal features to enhance predictive performance, with limited consideration on real-world clinical applicability, particularly in longitudinal NAT scenarios with multi-modal data. Here, we propose the Multi-modal Response Prediction (MRP) system, designed to mimic real-world physician assessments of NAT responses in breast cancer. To enhance feasibility, MRP integrates cross-modal knowledge mining and temporal information embedding strategy to handle missing modalities and remain less affected by different NAT settings. We validated MRP through multi-center studies and multinational reader studies. MRP exhibited comparable robustness to breast radiologists, outperforming humans in predicting pathological complete response in the Pre-NAT phase (ΔAUROC 14% and 10% on in-house and external datasets, respectively). Furthermore, we assessed MRP’s clinical utility impact on treatment decision-making. MRP may have profound implications for enrolment into NAT trials and determining surgery extensiveness.

Date: 2024
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
https://www.nature.com/articles/s41467-024-53450-8 Abstract (text/html)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:nat:natcom:v:15:y:2024:i:1:d:10.1038_s41467-024-53450-8

Ordering information: This journal article can be ordered from
https://www.nature.com/ncomms/

DOI: 10.1038/s41467-024-53450-8

Access Statistics for this article

Nature Communications is currently edited by Nathalie Le Bot, Enda Bergin and Fiona Gillespie

More articles in Nature Communications from Nature
Bibliographic data for series maintained by Sonal Shukla () and Springer Nature Abstracting and Indexing ().