How generative AIs support selection and evaluation in complex decision tasks: insights from academic paper review

Hao Yu, Ye Hou, Yuxian Liu and Yuan Li

Journal of Management Analytics, 2025, vol. 12, issue 3, 435-449

Abstract: This study investigates the role of generative large language models (GLLMs) in supporting complex selection and evaluation tasks within the academic paper review process. Using empirical data from management journal submissions, we compared the performance of six leading GLLMs (Claude 3.5, GPT-4O, Gemini 2.5, Deepseek-R3, Moonshot-V1 (kimi), and Qwen-Long) against human editors and reviewers. The results show that, at the editorial screening stage, GLLMs can help editors identify manuscripts with low publication potential, with aggregated model scores closely matching human editorial decisions. At the review stage, comments generated by the union of any three GLLMs from six GLLMs can cover over 61% of issues raised by human reviewers and are rated as superior by management professors. These findings demonstrate that GLLMs can complement human judgment in multi-stage, knowledge-intensive decision processes, improving both the efficiency and quality of academic paper reviews. The study expands the application boundaries of generative AI in management research evaluation and offers practical insights for integrating GLLMs into scholarly review workflows.

Date: 2025
References: Add references at CitEc
Citations:

Downloads: (external link)
http://hdl.handle.net/10.1080/23270012.2025.2537410 (text/html)
Access to full text is restricted to subscribers.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:taf:tjmaxx:v:12:y:2025:i:3:p:435-449

Ordering information: This journal article can be ordered from
http://www.tandfonline.com/pricing/journal/tjma20

DOI: 10.1080/23270012.2025.2537410

Access Statistics for this article

Journal of Management Analytics is currently edited by Li Xu

More articles in Journal of Management Analytics from Taylor & Francis Journals
Bibliographic data for series maintained by Chris Longhurst ().