Knowing (Not) to Know: Explainable Artificial Intelligence and Human Metacognition

Moritz von Zahn, Lena Liebich, Ekaterina Jussupow, Oliver Hinz and Kevin Bauer

Publications of Darmstadt Technical University, Institute for Business Studies (BWL) from Darmstadt Technical University, Department of Business Administration, Economics and Law, Institute for Business Studies (BWL)

Abstract: The use of explainable artificial intelligence (XAI) to render black-box artificial intelligence (AI) more interpretable to humans is gaining practical relevance. Prior research has shown that XAI can influence how humans “think”. Yet, little is known about whether XAI also affects how people “think about their thinking” (i.e., their metacognitive processes). We address this gap by investigating whether XAI affects metacognitive processes that guide human confidence judgments about their ability to perform a task and thereby, their decision whether to delegate the task to AI. We conducted two incentivized experiments in which domain experts repeatedly performed prediction tasks, with the option to delegate each task to an AI. We exogenously varied whether participants initially received explanations about AI’s overall prediction logic. We find that AI explanations improve how well humans understand their own ability to perform the task (metacognitive accuracy). This improvement causally increases both the frequency and effectiveness of human-to-AI delegation. Additional analyses show that these effects primarily occur when explanations reveal to humans that AI’s prediction logic diverges from their own, leading to a systematic reduction of overconfidence. Our findings highlight metacognitive processes as a central, previously overlooked channel through which AI explanations can influence human-AI collaboration. We discuss practical implications of our results for organizations implementing XAI to comply with regulatory transparency requirements as, for example, outlined in the European Union Artificial Intelligence Act.

Date: 2025-12-08
Note: for complete metadata visit http://tubiblio.ulb.tu-darmstadt.de/158974/
References: Add references at CitEc
Citations:

Published in Information System Research (2025-12-08)

There are no downloads for this item, see the EconPapers FAQ for hints about obtaining it.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:dar:wpaper:158974

Access Statistics for this paper

More papers in Publications of Darmstadt Technical University, Institute for Business Studies (BWL) from Darmstadt Technical University, Department of Business Administration, Economics and Law, Institute for Business Studies (BWL) Contact information at EDIRC.
Bibliographic data for series maintained by Dekanatssekretariat ().