Ordinal Response Elicitation for Systematic Review Screening with GPT

Ronald Luo, Abu Ilius Faisal, Ziya Sastimoglu and Jamal Deen

No tvekx_v1, MetaArXiv from Center for Open Science

Abstract: Background: Systematic reviews are crucial for evidence-based practice but are often hindered by labor-intensive methodologies. Recent advances in Large Language Models (LLMs) may help improve review efficiency and replicability. Objective: Building on prior research, this study evaluates the impact of prompt engineering on LLM performance in automating systematic reviews, specifically examining the "zero-shot" capability for predicting article inclusion based on titles and abstracts alone. Methods: We reanalyzed 24,534 studies across three systematic reviews using two sets of prompts with GPT-3.5 Turbo. One set rated the suitability of a study on a scale of one to ten, while the other elicited a binary response. Both ordinal-response and binary-response prompts were assessed for accuracy, sensitivity, specificity, predictive values, F1 score, and Matthews Correlation Coefficient (MCC). We also evaluated Receiver Operating Characteristic (ROC) curves for ordinal-response prompts and statistically validated the results using K-fold permutations and one-way analysis of variance. Results: Ordinal prompting allowed for fine-tuning of sensitivity and specificity, proving to be a viable zero-shot inclusion prompt with ROC Area Under Curve (AUC) scores between 80-90%. Including studies rated seven or higher achieved sensitivity and specificity of ~80%, while using the mean score increased sensitivity to nearly 100%, a task not possible with earlier strategies. Conclusion: Prompt engineering offers a scalable solution that allows researchers to target highly relevant articles or maximize sensitivity to include nearly all relevant studies if more time is available.

Date: 2026-07-31
References: Add references at CitEc
Citations:

Downloads: (external link)
https://osf.io/download/6a6ce8c9461a0a2026cf6ad8/

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:osf:metaar:tvekx_v1

DOI: 10.31219/osf.io/tvekx_v1

Access Statistics for this paper

More papers in MetaArXiv from Center for Open Science
Bibliographic data for series maintained by OSF ().