Evaluation of the accuracy and robustness of large language models in dental examinations: sensitivity analysis of question format and evaluation parameters

Andrea Atassi, Shachar Stewart, Carrie Brackenridge and Eliza Kamin

SAP Artificial Intelligence in Dentistry, 2026

Abstract: Introduction: Large language models are increasingly evaluated on medical examination questions, but their performance on dental questions, and the robustness of that performance to question format and decoding settings, is not established. Methods: A fixed set of 100 dental multiple-choice questions was drawn from the MedMCQA validation split and posed to a large language model in a zero-shot setting. Accuracy was scored against the reference answer, and sensitivity analyses varied question format, decoding temperature and sample size. Results: The model answered 70 of 100 dental examination questions correctly, for an accuracy of 0.700 (95% CI 0.604 to 0.781). The estimate remained essentially unchanged at a higher decoding temperature (0.690) and on a 50-question subsample (0.700), while it differed between single-choice (0.733) and multiple-choice (0.625) question formats. Conclusions: The model achieved a robust accuracy of 0.700 on dental examination questions, and the estimate was stable to temperature and sample size but sensitive to question format, with higher accuracy on single-choice than on multiple-choice questions.

Date: 2026
References: Add references at CitEc
Citations:

Downloads: (external link)
https://southam.pub/journals/files/aid/aid2026135en.pdf (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:cwf:aidart:aid2026135

DOI: 10.62486/aid2026135

Access Statistics for this article

More articles in SAP Artificial Intelligence in Dentistry from South American Publishing
Bibliographic data for series maintained by South American Publishing Journals Manager ().