Automating Evidence Synthesis: A Comparative Evaluation of Large Language Models for Data Extraction

Aditya Retnanto, Yohan Iddawela and Elaine Tan
Additional contact information
Aditya Retnanto: Asian Development Bank
Yohan Iddawela: Asian Development Bank
Elaine Tan: Asian Development Bank

No 845, ADB Economics Working Paper Series from Asian Development Bank

Abstract: Systematic reviews and meta-analyses (SRMAs) are important tools for evidence synthesis but have historically required substantial manual effort, particularly during the data extraction phase. To address this bottleneck, we developed and evaluated an automated pipeline that utilizes large language models (LLMs) to ingest full text scientific articles and extract structured metadata. We benchmarked the performance of leading models, including Gemini 2.5 Pro, GPT-5, and Sonnet 4.0, across two distinct domains: mobile health interventions and education. Our results indicate that Gemini 2.5 Pro achieved the strongest performance in qualitative metadata extraction and outcome identification. However, quantitative metadata extraction remained a significant challenge. Models struggled to interpret complex data across multiple tables and failed to calculate effect sizes when only raw figures were reported. Crucially, we find that human annotators often applied implicit filtering criteria not documented in the coding manual, which made benchmarking the results challenging. We discuss the implications of these findings, emphasizing that while LLMs can accelerate the coding process, reliable automation requires significantly more prescriptive coding manuals to strictly steer model behavior and ensure fair benchmarking.

Keywords: evidence synthesis automation; large language models (LLMs); data extraction benchmarking; systematic reviews and meta-analyses (SRMA) (search for similar items in EconPapers)
JEL-codes: C88 (search for similar items in EconPapers)
Pages: 34
Date: 2026-05-15
References: Add references at CitEc
Citations:

Downloads: (external link)
https://www.adb.org/publications/automating-evidence-synthesis Full text
None

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:ris:adbewp:022484

Access Statistics for this paper

More papers in ADB Economics Working Paper Series from Asian Development Bank 6 ADB Avenue, Mandaluyong City, 1550 Metro Manila, Philippines. Contact information at EDIRC.
Bibliographic data for series maintained by Orlee Velarde ().