Validation of a Natural Language Machine Learning Model for Safety Literature Surveillance

Jiyoon Park, Malek Djelassi, Daniel Chima, Robert Hernandez, Vladimir Poroshin, Ana-Maria Iliescu, Douglas Domalik and Noel Southall ()
Additional contact information
Jiyoon Park: AstraZeneca
Malek Djelassi: AstraZeneca
Daniel Chima: AstraZeneca
Robert Hernandez: AstraZeneca
Vladimir Poroshin: AstraZeneca
Ana-Maria Iliescu: AstraZeneca
Douglas Domalik: AstraZeneca
Noel Southall: AstraZeneca

Drug Safety, 2024, vol. 47, issue 1, No 6, 80 pages

Abstract: Abstract Introduction As part of routine safety surveillance, thousands of articles of potential interest are manually triaged for review by safety surveillance teams. This manual triage task is an interesting candidate for automation based on the abundance of process data available for training, the performance of natural language processing algorithms for this type of cognitive task, and the small number of safety signals that originate from literature review, resulting in its lower risk profile. However, deep learning algorithms introduce unique risks and the validation of such models for use in Good Pharmacovigilance Practice remains an open question. Objective Qualifying an automated, deep learning approach to literature surveillance for use at AstraZeneca. Methods The study is a prospective validation of a literature surveillance triage model, comparing its real-world performance with that of human surveillance teams working in parallel. The biggest risk in modifying this triage process is missing a safety signal (resulting in model false negatives) and hence model recall is the main evaluation metric considered. Results The model demonstrates consistent global performance from training through testing, with recall rates comparable to that of existing surveillance teams. The model is accepted for use specifically for those products where non-inferiority to the manual process is rigorously demonstrated. Conclusion Characterizing model performance prospectively, under real-world conditions, allows us to thoroughly examine model consistency and failure modes, qualifying it for use in our surveillance processes. We also identify potential future improvements and recognize the opportunity for the community to collaborate on this shared task.

Date: 2024
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
http://link.springer.com/10.1007/s40264-023-01367-4 Abstract (text/html)
Access to the full text of the articles in this series is restricted.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:spr:drugsa:v:47:y:2024:i:1:d:10.1007_s40264-023-01367-4

Ordering information: This journal article can be ordered from
http://www.springer.com/adis/journal/40264

DOI: 10.1007/s40264-023-01367-4

Access Statistics for this article

Drug Safety is currently edited by Nitin Joshi

More articles in Drug Safety from Springer
Bibliographic data for series maintained by Sonal Shukla () and Springer Nature Abstracting and Indexing ().