The use of large language models in generating multiple choice questions for health professions education: A systematic review and network meta-analysis

Lauren Riehm, Keean Nanji, Moiz Lakhani, Evelina Pankiv, Dean Hasanee and Wesla Pfeifer

PLOS ONE, 2026, vol. 21, issue 1, 1-19

Abstract: Purpose: Large language models (LLMs) have the potential to change medical education. Whether LLMs can generate multiple-choice questions (MCQs) that are of similar quality to those created by humans is unclear. This investigation assessed the quality of MCQs generated by LLMs compared to humans. Methods: This review was registered with PROSPERO (CRD42025608775). A systematic review and frequentist random-effects network meta-analysis (NMA) or pairwise meta-analysis was performed. Ovid MEDLINE, Ovid EMBASE, and Scopus were searched from inception to November 1, 2024. The quality of MCQs was assessed with seven pre-defined outcomes: question relevance, clarity, accuracy/correctness; distractor quality; item difficulty analysis; and item discrimination analysis (point biserial correlation and item discrimination index). Continuous data were transformed to a 10-point scale to facilitate statistical analysis and reported as mean differences (MD). The MERSQI and the Grade of Recommendations, Assessment, Development and Evaluation (GRADE) NMA guidelines were used to assess risk of bias and certainty of evidence assessments. Results: Five LLMs were included. NMA demonstrated that ChatGPT 4 generated similar quality MCQs to humans with regards to question relevance (MD −0.13; 95% CI: −0.44,0.18; GRADE: VERY LOW), question clarity (MD −0.03; 95% CI: −0.15,0.10; GRADE: VERY LOW), and distractor quality (MD −0.10; 95% CI: −0.24,0.04; GRADE: VERY LOW); however, MCQs generated by Llama 2 performed worse than humans with regards to question clarity (MD −1.21; 95% CI: −1.60,-0.82; GRADE: VERY LOW) and distractor quality (MD −1.50; 95% CI: −2.03,-0.97; GRADE: VERY LOW). Exploratory post-hoc t-tests demonstrated that ChatGPT 3.5 performed worse than Llama 2 and ChatGPT 4 with regards to question clarity and distractor quality (p

Date: 2026
References: Add references at CitEc
Citations:

Downloads: (external link)
https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0340277 (text/html)
https://journals.plos.org/plosone/article/file?id= ... 40277&type=printable (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:plo:pone00:0340277

DOI: 10.1371/journal.pone.0340277

Access Statistics for this article

More articles in PLOS ONE from Public Library of Science
Bibliographic data for series maintained by plosone ().