Man vs. machine: Experimental evidence on the quality and perceptions of AI-generated research content

Clemens Breisinger, Naureen Karachiwalla, Michael Keenan, MinAh Kim, Jawoo Koo and Christine Mwangi

No 2321, IFPRI discussion papers from International Food Policy Research Institute (IFPRI)

Abstract: Academic researchers want their research to be understood and used by non-technical audiences, but that requires communication that is more accessible in the form of non-technical and shorter summaries. The researcher must both signal the quality of the research and ensure that the content is salient by making it more readable. AI tools can improve salience; however, they can also lead to ambiguity in the signal since true effort is then difficult to observe. We implement an online factorial experiment providing non-technical audiences with a blog on an academic paper and vary the actual author of the blog from the same paper (human or ChatGPT) and whether respondents are told the blog is written by a human or AI tool. Even though AI-generated blogs are objectively of higher quality, they are rated lower, but not if the author is disclosed as AI, indicating that signaling is important and can be distorted by AI. Use of the blog does not vary by experimental arm. The findings suggest that, provided disclosure statements are included, researchers can potentially use AI to reduce effort costs without compromising signaling or salience. Academic researchers want their research to be understood and used by non-technical audiences, but that requires communication that is more accessible in the form of non-technical and shorter summaries. The researcher must both signal the quality of the research and ensure that the content is salient by making it more readable. AI tools can improve salience; however, they can also lead to ambiguity in the signal since true effort is then difficult to observe. We implement an online factorial experiment providing non-technical audiences with a blog on an academic paper and vary the actual author of the blog from the same paper (human or ChatGPT) and whether respondents are told the blog is written by a human or AI tool. Even though AI-generated blogs are objectively of higher quality, they are rated lower, but not if the author is disclosed as AI, indicating that signaling is important and can be distorted by AI. Use of the blog does not vary by experimental arm. The findings suggest that, provided disclosure statements are included, researchers can potentially use AI to reduce effort costs without compromising signaling or salience.

Keywords: artificial intelligence; communication; research; Southern Asia (search for similar items in EconPapers)
Date: 2024
New Economics Papers: this item is included in nep-ain and nep-exp
References: Add references at CitEc
Citations:

Downloads: (external link)
https://cgspace.cgiar.org/bitstreams/cc7599b1-02e9 ... 70a5e884e55/download (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:fpr:ifprid:2321

Access Statistics for this paper

More papers in IFPRI discussion papers from International Food Policy Research Institute (IFPRI) Contact information at EDIRC.
Bibliographic data for series maintained by ().