Artificial Intelligence to Automate Network Meta-Analyses: Four Case Studies to Evaluate the Potential Application of Large Language Models

Tim Reason (), Emma Benbow, Julia Langham, Andy Gimblett, Sven L. Klijn and Bill Malcolm
Additional contact information
Tim Reason: Estima Scientific, Mediaworks
Emma Benbow: Estima Scientific, Mediaworks
Julia Langham: Estima Scientific, Mediaworks
Andy Gimblett: Estima Scientific, Mediaworks
Sven L. Klijn: Bristol Myers Squibb
Bill Malcolm: Bristol Myers Squibb

PharmacoEconomics - Open, 2024, vol. 8, issue 2, No 4, 205-220

Abstract: Abstract Background The emergence of artificial intelligence, capable of human-level performance on some tasks, presents an opportunity to revolutionise development of systematic reviews and network meta-analyses (NMAs). In this pilot study, we aim to assess use of a large-language model (LLM, Generative Pre-trained Transformer 4 [GPT-4]) to automatically extract data from publications, write an R script to conduct an NMA and interpret the results. Methods We considered four case studies involving binary and time-to-event outcomes in two disease areas, for which an NMA had previously been conducted manually. For each case study, a Python script was developed that communicated with the LLM via application programming interface (API) calls. The LLM was prompted to extract relevant data from publications, to create an R script to be used to run the NMA and then to produce a small report describing the analysis. Results The LLM had a > 99% success rate of accurately extracting data across 20 runs for each case study and could generate R scripts that could be run end-to-end without human input. It also produced good quality reports describing the disease area, analysis conducted, results obtained and a correct interpretation of the results. Conclusions This study provides a promising indication of the feasibility of using current generation LLMs to automate data extraction, code generation and NMA result interpretation, which could result in significant time savings and reduce human error. This is provided that routine technical checks are performed, as recommend for human-conducted analyses. Whilst not currently 100% consistent, LLMs are likely to improve with time.

Date: 2024
References: View complete reference list from CitEc
Citations:

Downloads: (external link)
http://link.springer.com/10.1007/s41669-024-00476-9 Abstract (text/html)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:spr:pharmo:v:8:y:2024:i:2:d:10.1007_s41669-024-00476-9

Ordering information: This journal article can be ordered from
http://www.springer.com/adis/journal/41669

DOI: 10.1007/s41669-024-00476-9

Access Statistics for this article

PharmacoEconomics - Open is currently edited by Timothy Wrightson and Christopher Carswell

More articles in PharmacoEconomics - Open from Springer
Bibliographic data for series maintained by Sonal Shukla () and Springer Nature Abstracting and Indexing ().