PreRadE: Pretraining Tasks on Radiology Images and Reports Evaluation Framework

Matthew Coleman, Joanna F. Dipnall, Myong Chol Jung and Lan Du ()
Additional contact information
Matthew Coleman: Faculty of Information Technology, Monash University, Melbourne, VIC 3800, Australia
Joanna F. Dipnall: School of Public Health and Preventive Medicine, Monash University, Melbourne, VIC 38004, Australia
Myong Chol Jung: Faculty of Information Technology, Monash University, Melbourne, VIC 3800, Australia
Lan Du: Faculty of Information Technology, Monash University, Melbourne, VIC 3800, Australia

Mathematics, 2022, vol. 10, issue 24, 1-14

Abstract: Recently, self-supervised pretraining of transformers has gained considerable attention in analyzing electronic medical records. However, systematic evaluation of different pretraining tasks in radiology applications using both images and radiology reports is still lacking. We propose PreRadE, a simple proof of concept framework that enables novel evaluation of pretraining tasks in a controlled environment. We investigated three most-commonly used pretraining tasks (MLM—Masked Language Modelling, MFR—Masked Feature Regression, and ITM—Image to Text Matching) and their combinations against downstream radiology classification on MIMIC-CXR, a medical chest X-ray imaging and radiology text report dataset. Our experiments in the multimodal setting show that (1) pretraining with MLM yields the greatest benefit to classification performance, largely due to the task-relevant information learned from the radiology reports. (2) Pretraining with only a single task can introduce variation in classification performance across different fine-tuning episodes, suggesting that composite task objectives incorporating both image and text modalities are better suited to generating reliably performant models.

Keywords: self-supervised learning; deep learning applications; multimodal; masked language modelling; model evaluation; X-ray analysis; computational radiology; pathoanatomical classification; machine learning (search for similar items in EconPapers)
JEL-codes: C (search for similar items in EconPapers)
Date: 2022
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
https://www.mdpi.com/2227-7390/10/24/4661/pdf (application/pdf)
https://www.mdpi.com/2227-7390/10/24/4661/ (text/html)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:gam:jmathe:v:10:y:2022:i:24:p:4661-:d:997951

Access Statistics for this article

Mathematics is currently edited by Ms. Emma He

More articles in Mathematics from MDPI
Bibliographic data for series maintained by MDPI Indexing Manager ().