A Hybrid Semi-Automated Workflow for Systematic and Literature Review Processes with Large Language Model Analysis

Anjia Ye (), Ananda Maiti (), Matthew Schmidt and Scott J. Pedersen
Additional contact information
Anjia Ye: School of Education, University of Tasmania, Launceston, TAS 7248, Australia
Ananda Maiti: School of Information Technology, Deakin University, Geelong, VIC 3221, Australia
Matthew Schmidt: School of Health Sciences, University of Tasmania, Sandy Bay, TAS 7005, Australia
Scott J. Pedersen: School of Education, University of Tasmania, Launceston, TAS 7248, Australia

Future Internet, 2024, vol. 16, issue 5, 1-22

Abstract: Systematic reviews (SRs) are a rigorous method for synthesizing empirical evidence to answer specific research questions. However, they are labor-intensive because of their collaborative nature, strict protocols, and typically large number of documents. Large language models (LLMs) and their applications such as gpt-4/ChatGPT have the potential to reduce the human workload of the SR process while maintaining accuracy. We propose a new hybrid methodology that combines the strengths of LLMs and humans using the ability of LLMs to summarize large bodies of text autonomously and extract key information. This is then used by a researcher to make inclusion/exclusion decisions quickly. This process replaces the typical manually performed title/abstract screening, full-text screening, and data extraction steps in an SR while keeping a human in the loop for quality control. We developed a semi-automated LLM-assisted (Gemini-Pro) workflow with a novel innovative prompt development strategy. This involves extracting three categories of information including identifier , verifier , and data field (IVD) from the formatted documents. We present a case study where our hybrid approach reduced errors compared with a human-only SR. The hybrid workflow improved the accuracy of the case study by identifying 6/390 (1.53%) articles that were misclassified by the human-only process. It also matched the human-only decisions completely regarding the rest of the 384 articles. Given the rapid advances in LLM technology, these results will undoubtedly improve over time.

Keywords: artificial intelligence; large language model; systematic review; methodology; workflow (search for similar items in EconPapers)
JEL-codes: O3 (search for similar items in EconPapers)
Date: 2024
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
https://www.mdpi.com/1999-5903/16/5/167/pdf (application/pdf)
https://www.mdpi.com/1999-5903/16/5/167/ (text/html)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:gam:jftint:v:16:y:2024:i:5:p:167-:d:1393142

Access Statistics for this article

Future Internet is currently edited by Ms. Grace You

More articles in Future Internet from MDPI
Bibliographic data for series maintained by MDPI Indexing Manager ().