Model-agnostic auditing: a lost cause?

Sakina Hansen and Joshua Loftus

LSE Research Online Documents on Economics from London School of Economics and Political Science, LSE Library

Abstract: Tools for interpretable machine learning (IML) or explainable artificial intelligence (xAI) can be used to audit algorithms for fairness or other desiderata. In a black-box setting without access to the algorithm’s internal structure an auditor may be limited to methods that are model-agnostic. These methods have severe limitations with important consequences for outcomes such as fairness. Among model-agnostic IML methods, visualizations such as the partial dependence plot (PDP) or individual conditional expectation (ICE) plots are popular and useful for displaying qualitative relationships. Although we focus on fairness auditing with PDP/ICE plots, the consequences we highlight generalize to other auditing or IML/xAI applications. This paper questions the validity of auditing in high-stakes settings with contested values or conflicting interests if the audit methods are model-agnostic.

Keywords: artificial intelligence; black-box auditing; causal models; CEUR Workshop Proceedings (CEUR-WS.org); counterfactual fairness; individual conditional expectation; machine learning; partial dependence plots; supervised learning; visualization (search for similar items in EconPapers)
JEL-codes: C1 (search for similar items in EconPapers)
Date: 2023-07-16
New Economics Papers: this item is included in nep-ain, nep-big, nep-cmp and nep-ecm
References: View references in EconPapers View complete reference list from CitEc
Citations:

Published in CEUR Workshop Proceedings, 16, July, 2023, 3442. ISSN: 1613-0073

Downloads: (external link)
http://eprints.lse.ac.uk/120114/ Open access version. (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:ehl:lserod:120114

Access Statistics for this paper

More papers in LSE Research Online Documents on Economics from London School of Economics and Political Science, LSE Library LSE Library Portugal Street London, WC2A 2HD, U.K.. Contact information at EDIRC.
Bibliographic data for series maintained by LSERO Manager ().