On the Use of Outcome Tests for Detecting Bias in Decision Making

Ivan Canay, Magne Mogstad and Jack Mountjoy

No 2020-125, Working Papers from Becker Friedman Institute for Research In Economics

Abstract: The decisions of judges, lenders, journal editors, and other gatekeepers often lead to disparities in outcomes across affected groups. An important question is whether, and to what extent, these group-level disparities are driven by relevant differences in underlying individual characteristics, or by biased decision makers. Becker (1957) proposed an outcome test for bias leading to a large body of related empirical work, with recent innovations in settings where decision makers are exogenously assigned to cases and vary progressively in their decision tendencies. We carefully examine what can be learned about bias in decision making in such settings. Our results call into question recent conclusions about racial bias among bail judges, and, more broadly, yield four lessons for researchers considering the use of outcome tests of bias. First, the so-called generalized Roy model, which is a workhorse of applied economics, does not deliver a logically valid outcome test without further restrictions, since it does not require an unbiased decision maker to equalize marginal outcomes across groups. Second, the more restrictive "extended" Roy model, which isolates potential outcomes as the sole admissible source of analyst-unobserved variation driving decisions, delivers both a logically valid and econometrically viable outcome test. Third, this extended Roy model places strong restrictions on behavior and the data generating process, so detailed institutional knowledge is essential for justifying such restrictions. Finally, because the extended Roy model imposes restrictions beyond those required to identify marginal outcomes across groups, it has testable implications that may help assess its suitability across empirical settings.

Keywords: Outcome test; bias; discrimination; generalized Roy model; extended Roy model; marginal treatment effect; judge leniency; bail; pre-trial release (search for similar items in EconPapers)
Pages: 45 pages
Date: 2020
New Economics Papers: this item is included in nep-law
References: View references in EconPapers View complete reference list from CitEc
Citations: View citations in EconPapers (7)

Downloads: (external link)
https://repec.bfi.uchicago.edu/RePEc/pdfs/BFI_WP_2020125.pdf (application/pdf)

Related works:
Journal Article: On the Use of Outcome Tests for Detecting Bias in Decision Making (2024)
Working Paper: On the Use of Outcome Tests for Detecting Bias in Decision Making (2020)
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:bfi:wpaper:2020-125

Access Statistics for this paper

More papers in Working Papers from Becker Friedman Institute for Research In Economics Contact information at EDIRC.
Bibliographic data for series maintained by Toni Shears ( this e-mail address is bad, please contact ).