Can we trust LLMs to help us? An examination of the potential use of GPT-4 in generating quality literature reviews

Min Zhao, Fuan Li, Francis Cai, Haiyang Chen and Zheng Li

Nankai Business Review International, 2024, vol. 16, issue 1, 128-142

Abstract: Purpose - This study aims to examine the ability of Generative Pre-trained Transformer 4 (GPT-4), one of the most powerful large language models, to generate a literature review for peer-reviewed journal publications. The objective is to determine whether business scholars can rely on GPT-4’s assistance with literature reviews and how the nature of human–artificial intelligence (AI) interaction may affect the quality of the reviews generated by GPT-4. Design/methodology/approach - A survey of 30 experienced researchers was conducted to assess the quality of the literature reviews generated by GPT-4 in comparison with a human-authored literature review published in a Social Science Citation Index (SSCI) journal. The data collected were then analyzed with analysis of variance to ascertain whether we may trust GPT-4’s assistance in writing literature reviews. Findings - The statistical analysis reveals that when a highly structured approach being used, GPT-4 can generate a high-quality review comparable to that found in an SSCI journal publication. However, when a less structured approach is used, the generated review lacks comprehensive understating and critical analysis, and is unable to identify literature gaps for future research, although it performed well in adequate synthesis and quality writing. The findings suggest that we may trust GPT-4 to generate literature reviews that align with the publication standards of a peer-reviewed journal when using a structured approach to human–AI interaction. Research limitations/implications - The findings suggest that we may trust GPT-4 to generate literature reviews that align with the publication standards of a peer-reviewed journal when using a structured approach to human–AI interaction. Nonetheless, cautions should be taken due to the limitations of this study discussed in the text. Originality/value - By breaking down the specific tasks of a literature review and using a quantitative rather than qualitative assessment method, this study provides robust and more objective findings about the ability of GPT-4 to assist us with a very important research task. The findings of this study should enhance our understanding of how GPT-4 may change our research endeavor and how we may take a full advantage of the advancement in AI technology in the future research.

Keywords: Literature reviews; Large language models; GPT-4; Structured human-AI interaction (search for similar items in EconPapers)
Date: 2024
References: Add references at CitEc
Citations:

Downloads: (external link)
https://www.emerald.com/insight/content/doi/10.110 ... d&utm_campaign=repec (text/html)
https://www.emerald.com/insight/content/doi/10.110 ... d&utm_campaign=repec (application/pdf)
Access to full text is restricted to subscribers

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:eme:nbripp:nbri-12-2023-0115

DOI: 10.1108/NBRI-12-2023-0115

Access Statistics for this article

Nankai Business Review International is currently edited by Dr Xuexiu Wang and Professor Li Wei'an

More articles in Nankai Business Review International from Emerald Group Publishing Limited
Bibliographic data for series maintained by Emerald Support ().