Mirror, Mirror on the Wall: Algorithmic Assessments, Transparency, and Self-Fulfilling Prophecies

Kevin Bauer () and Andrej Gill ()
Additional contact information
Kevin Bauer: Information Systems Department, University of Mannheim, 68161 Mannheim, Germany
Andrej Gill: Economics and Management, Johannes Gutenberg-University, 55128 Mainz, Germany

Information Systems Research, 2024, vol. 35, issue 1, 226-248

Abstract: Predictive algorithmic scores can significantly impact the lives of assessed individuals by shaping decisions of organizations and institutions that affect them, for example, influencing the hiring prospects of job applicants or the release of defendants on bail. To better protect people and provide them the opportunity to appeal their algorithmic assessments, data privacy advocates and regulators increasingly push for disclosing the scores and their use in decision-making processes to scored individuals. Although inherently important, the response of scored individuals to such algorithmic transparency is understudied and therefore demands further research. Inspired by psychological and economic theories of information processing, we aim to fill this gap. We conducted a comprehensive experimental study with five treatment conditions to explore how and why disclosing the use of algorithmic scoring processes to (involuntarily) scored individuals affects their behaviors. Our results provide strong evidence that the disclosure of fundamentally erroneous algorithmic scores evokes self-fulfilling prophecies that endogenously steer the behavior of scored individuals toward their assessment, enabling algorithms to help produce the world they predict. Occurring self-fulfilling prophecies are consistent with an anchoring effect and the exploitation of available moral wiggle room. Because scored individuals interpret others’ motives for overriding human expert and algorithmic scores differently, self-fulfilling prophecies occur in part only when disclosing algorithmic scores. Our results emphasize that isolated transparency measures can have considerable side effects with noticeable implications for the development of automation bias, the occurrence of feedback loops, and the design of transparency regulations.

Keywords: algorithmic transparency; algorithmic scoring; self-fulfilling prophecies; second-order beliefs; algorithm aversion (search for similar items in EconPapers)
Date: 2024
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
http://dx.doi.org/10.1287/isre.2023.1217 (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:inm:orisre:v:35:y:2024:i:1:p:226-248

Access Statistics for this article

More articles in Information Systems Research from INFORMS Contact information at EDIRC.
Bibliographic data for series maintained by Chris Asher ().