Designing Human-AI Collaboration: A Sufficient-Statistic Approach

Nikhil Agarwal, Alex Moehring and Alexander Wolitzky

No 33949, NBER Working Papers from National Bureau of Economic Research, Inc

Abstract: We develop a sufficient-statistic approach to designing collaborative human-AI decision-making policies in classification problems, where AI predictions can be used to either automate decisions or selectively assist humans. The approach allows for endogenous and biased beliefs, and effort crowd-out, without imposing a structural model of human decision-making. We deploy and validate our approach in an online fact-checking experiment. We find that humans under-respond to AI predictions and reduce effort when presented with confident AI predictions. AI under-response stems more from human overconfidence in own-signal precision than from under-confidence in AI. The optimal policy automates cases where the AI is confident and delegates uncertain cases to humans while fully disclosing the AI prediction. While both automation and human judgement are valuable, the incremental benefit over selective automation of assisting humans with AI predictions is negligible. The sufficient-statistic approach accurately predicts the performance of out-of-sample collaboration policies, suggesting it can be a useful guide to designing collaborative systems.

JEL-codes: C91 D47 D83 D89 (search for similar items in EconPapers)
Date: 2025-06
New Economics Papers: this item is included in nep-ain and nep-exp
Note: IO PR
References: Add references at CitEc
Citations: View citations in EconPapers (4)

Downloads: (external link)
http://www.nber.org/papers/w33949.pdf (application/pdf)
Access to the full text is generally limited to series subscribers, however if the top level domain of the client browser is in a developing country or transition economy free access is provided. More information about subscriptions and free access is available at http://www.nber.org/wwphelp.html. Free access is also available to older working papers.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:nbr:nberwo:33949

Ordering information: This working paper can be ordered from
http://www.nber.org/papers/w33949
The price is Paper copy available by mail.

Access Statistics for this paper

More papers in NBER Working Papers from National Bureau of Economic Research, Inc National Bureau of Economic Research, 1050 Massachusetts Avenue Cambridge, MA 02138, U.S.A.. Contact information at EDIRC.
Bibliographic data for series maintained by ().