AI versus human coding of NIH grant abstracts

Sarah A Alkhatib, Nadiya Jiwa, Dallin Judd, Justin M Luningham, Ginnie Sawyer-Morris, Merve Ulukaya, Todd Molfenter, Faye S Taxman and Scott T Walters

PLOS ONE, 2026, vol. 21, issue 10, 1-11

Abstract: Large language models (LLMs) are increasingly used for qualitative analysis in substance use research, yet their performance relative to human coders remains underexplored. This study compares ChatGPT-4.0 with human coders in performing qualitative coding tasks using NIH grant abstracts as a test case, focusing on the identification and description of research innovations. Using a sample of NIH HEAL Initiative grant abstracts related to opioid overdose prevention, a total of 125 abstracts were independently coded by ChatGPT and humans to generate innovation descriptions, which were then evaluated by both human raters and ChatGPT for depth/detail and relevance/completeness using 5-point Likert scales. Identical instructions were used across all coding and evaluation stages. ChatGPT-generated descriptions were consistently rated higher than human-generated descriptions on both dimensions. Human evaluators rated ChatGPT outputs at an average of 4.47 for both depth/detail and relevance/completeness, compared to 3.33 and 3.24 for human outputs, respectively (F(1,176)=133.9, p

Date: 2026
References: Add references at CitEc
Citations:

Downloads: (external link)
https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0343857 (text/html)
https://journals.plos.org/plosone/article/file?id= ... 43857&type=printable (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:plo:pone00:0343857

DOI: 10.1371/journal.pone.0343857

Access Statistics for this article

More articles in PLOS ONE from Public Library of Science
Bibliographic data for series maintained by plosone ().