Can ChatGPT Evaluate Plans?

Xinyu Fu, Ruoniu Wang and Chaosu Li

Journal of the American Planning Association, 2024, vol. 90, issue 3, 525-536

Abstract: Problem, research strategy, and findingsLarge language models, such as ChatGPT, have recently risen to prominence in producing human-like conversation and assisting with various tasks, particularly for analyzing high-dimensional textual materials. Because planning researchers and practitioners often need to evaluate planning documents that are long and complex, a first-ever possible question has emerged: Can ChatGPT evaluate plans? In this study we addressed this question by leveraging ChatGPT to evaluate the quality of plans and compare the results with those conducted by human coders. Through the evaluation of 10 climate change plans, we discovered that ChatGPT’s evaluation results coincided reasonably well (with an average of 68%) with those from the traditional content analysis approach. We further scrutinized the differences by conducting a more in-depth analysis of the results from ChatGPT and manual evaluation to uncover what might have contributed to the variance in results. Our findings indicate that ChatGPT struggled to comprehend planning-specific jargon, yet it could reduce human errors by capturing details in complex planning documents. Finally, we provide insights into leveraging this cutting-edge technology in future planning research and practice.Takeaway for practiceChatGPT cannot be used to replace humans in plan quality evaluation yet. However, it is an effective tool to complement human coders to minimize human errors by identifying discrepancies and fact-checking machine-generated responses. ChatGPT generally cannot understand planning jargon, so planners wanting to use this tool should use extra caution when planning terminologies are present in their prompts. Creating effective prompts for ChatGPT is an iterative process that requires specific instructions.

Date: 2024
References: Add references at CitEc
Citations:

Downloads: (external link)
http://hdl.handle.net/10.1080/01944363.2023.2271893 (text/html)
Access to full text is restricted to subscribers.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:taf:rjpaxx:v:90:y:2024:i:3:p:525-536

Ordering information: This journal article can be ordered from
http://www.tandfonline.com/pricing/journal/rjpa20

DOI: 10.1080/01944363.2023.2271893

Access Statistics for this article

Journal of the American Planning Association is currently edited by Sandi Rosenbloom

More articles in Journal of the American Planning Association from Taylor & Francis Journals
Bibliographic data for series maintained by Chris Longhurst ().