Comparative analysis of concept mapping: human participants vs. ChatGPT

Stephen T. Homer ()
Additional contact information
Stephen T. Homer: Sunway University

Quality & Quantity: International Journal of Methodology, 2025, vol. 59, issue 5, No 40, 4873-4892

Abstract: Abstract ChatGPT has great potential in academia, but exploring AI’s impact on social sciences research has been neglected. This paper uses concept mapping, a systematic way to organize and represent group ideas, to evaluate ChatGPT and compare it to a human study. As ChatGPT collects ideas from a variety of human generated training data and presents them to the user. Thus, could ChatGPT falsify social science survey, questionnaire, and sorting results? The study employs a methodology centered around concept mapping, a bottom-up exploratory research design widely used in various fields, comprising five stages: statement generation, statement grouping, multi-dimensional scaling (MDS), hierarchical cluster analysis (HCA), and cluster labelling. However, to compare ChatGPT’s performance, adjustments are made. ChatGPT is prompted to thematically group provided statements and generate outputs. Despite being instructed to include all statements, ChatGPT’s iterations consistently miss some. These outputs are then entered into software for MDS and HCA analysis. Results demonstrate disparities in bridging values, indicating statement groupings’ coherence, reveal significant differences between ChatGPT and human-generated maps. These methodological differences reflect challenges in integrating AI technologies like ChatGPT into research methodologies. While ChatGPT’s natural language processing facilitates accessibility, its inability to consistently include all data and accurately replicate human cognitive processes poses limitations. Thus, rigorous evaluation and quality control procedures are imperative to ensure the reliability and accuracy of research findings when employing AI technologies in social science research.

Keywords: Group concept mapping; ChatGPT; Artificial intelligence; Research implications (search for similar items in EconPapers)
Date: 2025
References: Add references at CitEc
Citations:

Downloads: (external link)
http://link.springer.com/10.1007/s11135-025-02211-w Abstract (text/html)
Access to the full text of the articles in this series is restricted.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:spr:qualqt:v:59:y:2025:i:5:d:10.1007_s11135-025-02211-w

Ordering information: This journal article can be ordered from
http://www.springer.com/economics/journal/11135

DOI: 10.1007/s11135-025-02211-w

Access Statistics for this article

Quality & Quantity: International Journal of Methodology is currently edited by Vittorio Capecchi

More articles in Quality & Quantity: International Journal of Methodology from Springer
Bibliographic data for series maintained by Sonal Shukla () and Springer Nature Abstracting and Indexing ().