Pre-trained multimodal large language model enhances dermatological diagnosis using SkinGPT-4

Juexiao Zhou, Xiaonan He (), Liyuan Sun, Jiannan Xu, Xiuying Chen, Yuetan Chu, Longxi Zhou, Xingyu Liao, Bin Zhang, Shawn Afvari and Xin Gao ()
Additional contact information
Juexiao Zhou: King Abdullah University of Science and Technology (KAUST)
Xiaonan He: Affiliated to Capital Medical University
Liyuan Sun: Affiliated to Capital Medical University
Jiannan Xu: Affiliated to Capital Medical University
Xiuying Chen: King Abdullah University of Science and Technology (KAUST)
Yuetan Chu: King Abdullah University of Science and Technology (KAUST)
Longxi Zhou: King Abdullah University of Science and Technology (KAUST)
Xingyu Liao: King Abdullah University of Science and Technology (KAUST)
Bin Zhang: King Abdullah University of Science and Technology (KAUST)
Shawn Afvari: LLC
Xin Gao: King Abdullah University of Science and Technology (KAUST)

Nature Communications, 2024, vol. 15, issue 1, 1-12

Abstract: Abstract Large language models (LLMs) are seen to have tremendous potential in advancing medical diagnosis recently, particularly in dermatological diagnosis, which is a very important task as skin and subcutaneous diseases rank high among the leading contributors to the global burden of nonfatal diseases. Here we present SkinGPT-4, which is an interactive dermatology diagnostic system based on multimodal large language models. We have aligned a pre-trained vision transformer with an LLM named Llama-2-13b-chat by collecting an extensive collection of skin disease images (comprising 52,929 publicly available and proprietary images) along with clinical concepts and doctors’ notes, and designing a two-step training strategy. We have quantitatively evaluated SkinGPT-4 on 150 real-life cases with board-certified dermatologists. With SkinGPT-4, users could upload their own skin photos for diagnosis, and the system could autonomously evaluate the images, identify the characteristics and categories of the skin conditions, perform in-depth analysis, and provide interactive treatment recommendations.

Date: 2024
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
https://www.nature.com/articles/s41467-024-50043-3 Abstract (text/html)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:nat:natcom:v:15:y:2024:i:1:d:10.1038_s41467-024-50043-3

Ordering information: This journal article can be ordered from
https://www.nature.com/ncomms/

DOI: 10.1038/s41467-024-50043-3

Access Statistics for this article

Nature Communications is currently edited by Nathalie Le Bot, Enda Bergin and Fiona Gillespie

More articles in Nature Communications from Nature
Bibliographic data for series maintained by Sonal Shukla () and Springer Nature Abstracting and Indexing ().