Quality and Accountability of Large Language Models (LLMs) in Healthcare in Low- And Middle-Income Countries (LMIC): A Simulated Patient Study Using ChatGPT

Yafei Si (), Yuyi Yang, Xi Wang, Ruopeng An (), Jiaqi Zu, Xi Chen, Xiaojing Fan and Sen Gong
Additional contact information
Yafei Si: University of New South Wales
Yuyi Yang: Washington University, St. Louis
Xi Wang: Washington University, St. Louis
Ruopeng An: Washington University, St. Louis
Jiaqi Zu: Duke Kunshan University
Xiaojing Fan: Xi’an Jiaotong University
Sen Gong: Zhejiang University

No 17204, IZA Discussion Papers from Institute of Labor Economics (IZA)

Abstract: Using simulated patients to mimic nine established non-communicable and infectious diseases over 27 trials, we assess ChatGPT's effectiveness and reliability in diagnosing and treating common diseases in low- and middle-income countries. We find ChatGPT's performance varied within a single disease, despite a high level of accuracy in both correct diagnosis (74.1%) and medication prescription (84.5%). Additionally, ChatGPT recommended a concerning level of unnecessary or harmful medications (85.2%) even with correct diagnoses. Finally, ChatGPT performed better in managing non-communicable diseases compared to infectious ones. These results highlight the need for cautious AI integration in healthcare systems to ensure quality and safety.

Keywords: safety; quality; ChatGPT; Large Language Models; generative AI; simulated patient; healthcare; low- and middle-income countries (search for similar items in EconPapers)
JEL-codes: C0 C90 I10 I11 (search for similar items in EconPapers)
Pages: 11 pages
Date: 2024-08
New Economics Papers: this item is included in nep-hea
References: View complete reference list from CitEc
Citations:

Published - published as 'Quality and Accountability of ChatGPT in Health Care in Low- and Middle-Income Countries: Simulated Patient Study' in: Journal of Medical Internet Research, 2024, 26, e56121

Downloads: (external link)
https://docs.iza.org/dp17204.pdf (application/pdf)

Related works:
Working Paper: Quality and Accountability of Large Language Models (LLMs) in Healthcare in Low- and Middle-Income Countries (LMIC): A Simulated Patient Study using ChatGPT (2024)
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:iza:izadps:dp17204

Ordering information: This working paper can be ordered from
IZA, Margard Ody, P.O. Box 7240, D-53072 Bonn, Germany

Access Statistics for this paper

More papers in IZA Discussion Papers from Institute of Labor Economics (IZA) IZA, P.O. Box 7240, D-53072 Bonn, Germany. Contact information at EDIRC.
Bibliographic data for series maintained by Holger Hinte ().