Effectiveness of Transformer-Based Large Language Models in Identifying Adverse Drug Reaction Relations from Unstructured Discharge Summaries in Singapore

Yen Ling Koon, Yan Tung Lam, Hui Xing Tan, Desmond Hwee Chun Teo, Jing Wei Neo, Aaron Jun Yi Yap, Pei San Ang, Celine Ping Wei Loke, Mun Yee Tham, Siew Har Tan, Sally Leng Bee Soh, Belinda Qin Pei Foo, Zheng Jye Ling, James Luen Wei Yip and Sreemanee Raaj Dorajoo ()
Additional contact information
Yen Ling Koon: Health Sciences Authority
Yan Tung Lam: Health Sciences Authority
Hui Xing Tan: Health Sciences Authority
Desmond Hwee Chun Teo: Health Sciences Authority
Jing Wei Neo: Health Sciences Authority
Aaron Jun Yi Yap: Health Sciences Authority
Pei San Ang: Health Sciences Authority
Celine Ping Wei Loke: Health Sciences Authority
Mun Yee Tham: Health Sciences Authority
Siew Har Tan: Health Sciences Authority
Sally Leng Bee Soh: Health Sciences Authority
Belinda Qin Pei Foo: Health Sciences Authority
Zheng Jye Ling: National University of Singapore, National University Health System
James Luen Wei Yip: National University Heart Centre
Sreemanee Raaj Dorajoo: Health Sciences Authority

Drug Safety, 2025, vol. 48, issue 6, No 6, 667-677

Abstract: Abstract Introduction Transformer-based large language models (LLMs) have transformed the field of natural language processing and led to significant advancements in various text processing tasks. However, the applicability of these LLMs in identifying related drug-adverse event (AE) pairs within clinical context may be limited by the prevalent use of non-standard sentence structures and grammar. Method Nine transformer-based LLMs pre-trained on biomedical domain corpora are fine-tuned on annotated data (n = 5088) to classify drug-AE pairs in unstructured discharge summaries as causally related or unrelated. These LLMs are then validated on text segments from deidentified hospital discharge summaries from Singapore (n = 1647). To assess generalisability, the models are validated on annotated segments (n = 4418) from the Medical Information Mart for Intensive Care (MIMIC-III) database. Performance of LLMs in identifying related drug-AE pairs is then compared against a prior benchmark set by traditional machine learning models on the same data. Results Using an LLM-Bidirectional long short-term memory (LLM-BiLSTM) architecture, transformer-based LLMs improve F1 score as compared to prior benchmark with BioM-ELECTRA-Large-BiLSTM showing an average F1 score improvement of 16.1% (increase from 0.64 to 0.74). Applying additional rules on the LLM-based predictions, like ignoring drug-AE pairs when the AE is a known indication of the drug, results in a further reduction in false positive rates with precision increases of up to 5.6% (0.04 increment). Conclusion Transformer-based LLMs outperform traditional machine learning methods in identifying causally related drug-AE pairs embedded within unstructured discharge summaries. Nonetheless the improvement in performance with rules indicates that LLMs still possess some degree of imperfection for this causal relation detection task.

Date: 2025
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
http://link.springer.com/10.1007/s40264-025-01525-w Abstract (text/html)
Access to the full text of the articles in this series is restricted.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:spr:drugsa:v:48:y:2025:i:6:d:10.1007_s40264-025-01525-w

Ordering information: This journal article can be ordered from
http://www.springer.com/adis/journal/40264

DOI: 10.1007/s40264-025-01525-w

Access Statistics for this article

Drug Safety is currently edited by Nitin Joshi

More articles in Drug Safety from Springer
Bibliographic data for series maintained by Sonal Shukla () and Springer Nature Abstracting and Indexing ().