Unlocking the black box: Enhancing human-AI collaboration in high-stakes healthcare scenarios through explainable AI

Reda Hassan, Nhien Nguyen, Stine Rasdal Finserås, Lars Adde, Inga Strümke and Ragnhild Støen

Technological Forecasting and Social Change, 2025, vol. 219, issue C

Abstract: Despite the advanced predictive capabilities of artificial intelligence (AI) systems, their inherent opacity often leaves users confused about the rationale behind their outputs. We investigate the challenge of AI opacity, which undermines user trust and the effectiveness of clinical judgment in healthcare. We demonstrate how human experts make judgment in high-stakes scenarios where their judgment diverges from AI predictions, emphasizing the need for explainability to enhance clinical judgment and trust in AI systems. We used a scenario-based methodology, conducting 28 semi-structured interviews and observations with clinicians from Norway and Egypt. Our analysis revealed that, during the process of forming judgments, human experts engage in AI interrogation practices when faced with opaque AI systems. Obtaining explainability from AI systems leads to increased interrogation practices aimed at gaining a deeper understanding of AI predictions. With the introduction of explainable AI (XAI), experts demonstrate greater trust in the AI system, show a readiness to learn from AI, and may reconsider or update their initial judgments when they contradict AI predictions.

Keywords: Opacity; Explainable AI (XAI); Uncertainty; Professional judgment; Decision making; Medical diagnosis; Trust in AI (search for similar items in EconPapers)
Date: 2025
References: Add references at CitEc
Citations:

Downloads: (external link)
http://www.sciencedirect.com/science/article/pii/S0040162525002963
Full text for ScienceDirect subscribers only

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:eee:tefoso:v:219:y:2025:i:c:s0040162525002963

DOI: 10.1016/j.techfore.2025.124265

Access Statistics for this article

Technological Forecasting and Social Change is currently edited by Fred Phillips

More articles in Technological Forecasting and Social Change from Elsevier
Bibliographic data for series maintained by Catherine Liu ().