Augmenting patient safety surveillance in radiation oncology with large language model-based root cause analysis: A proof-of-concept study

Yuntao Wang, Mariluz De Ornelas, Matthew T Studenski, Elizabeth Bossart, Siamak P Nejad-Davarani and Yunze Yang

PLOS Digital Health, 2026, vol. 5, issue 9, 1-18

Abstract: We investigated the potential utility of large language models (LLMs) in supporting patient safety efforts. Specifically, we evaluated the reasoning capabilities of LLMs in performing root cause analysis (RCA) of radiation oncology incidents using narrative reports from the Radiation Oncology Incident Learning System (RO-ILS). We prompted four state-of-the-art LLMs, Gemini 2.5 Pro, GPT-4o, o3, and Grok 3, with the “Background and Incident Overview” sections from 19 publicly available RO-ILS cases. Each model was instructed to perform RCA and generate root causes, lessons learned, and suggested actions using a standardized prompt based on AAPM RCA guidelines. Model outputs were evaluated using a combination of objective semantic similarity metrics (cosine similarity via Sentence Transformer), semi-subjective assessments (precision, recall, F1-score, expert-adjudicated PPV (Positive Predictive Value), hallucination rate and performance criteria including relevance, comprehensiveness, quality of justification and quality of solution), and subjective ratings (reasoning quality and overall performance) by five board-certified medical physicists. LLMs demonstrated satisfactory performance across evaluation metrics. All models exhibited some degree of hallucination, ranging from 11% to 61%. All the evaluated LLMs demonstrated comparable baseline capabilities in objective causal extraction, and Gemini 2.5 Pro exhibited the highest overall performance score among 4 models. Statistically significant differences were observed among models in expert-adjudicated PPV, hallucination rate, and subjective ratings (p

Date: 2026
References: Add references at CitEc
Citations:

Downloads: (external link)
https://journals.plos.org/digitalhealth/article?id=10.1371/journal.pdig.0001740 (text/html)
https://journals.plos.org/digitalhealth/article/fi ... 01740&type=printable (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:plo:pdig00:0001740

DOI: 10.1371/journal.pdig.0001740

Access Statistics for this article

More articles in PLOS Digital Health from Public Library of Science
Bibliographic data for series maintained by digitalhealth ().