Observational vs. Experimental Data When Making Automated Decisions Using Machine Learning

Carlos Fernández-Loría () and Foster Provost ()
Additional contact information
Carlos Fernández-Loría: School of Business and Management, Hong Kong University of Science and Technology, New Territories, Hong Kong
Foster Provost: Leonard N. Stern School of Business, New York University, New York, New York 10012

INFORMS Joural on Data Science, 2025, vol. 4, issue 3, 197-229

Abstract: Decisions supported by machine learning often aim to improve outcomes through interventions, such as influencing purchasing behavior with ads or increasing customer retention with special offers. However, using observational data to estimate these effects can introduce confounding bias. Although experimental data can mitigate confounding, it is not always feasible to obtain and can be costly when it is. This paper presents theoretical results focusing on the impact of confounding on decision making, emphasizing that optimizing decisions often involves determining whether a causal effect exceeds a threshold rather than minimizing bias in the estimate. Consequently, models built with readily available but confounded data can sometimes yield decisions as good as or better than those based on costly, unconfounded data. This can occur when larger effects are more likely to be overestimated or when the benefits of larger, cheaper data sets outweigh the drawbacks of confounding. We validate the theoretical findings using benchmark data from the 2016 Atlantic Causal Inference Conference causal modeling competition, encompassing 77 scenarios and 7,700 data sets. We then introduce theoretical conditions, weaker than ignorability, that characterize when confounding preserves effect rankings. These conditions allow for empirical heuristic tests to assess whether observational data aligns with this structure. Finally, we apply our findings in a large-scale case study using advertising data, demonstrating how these insights can guide decision making in practice.

Keywords: causal inference; causal decision making; observational data; confounding bias; experimental data; treatment assignment (search for similar items in EconPapers)
Date: 2025
References: Add references at CitEc
Citations:

Downloads: (external link)
http://dx.doi.org/10.1287/ijds.2023.0012 (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:inm:orijds:v:4:y:2025:i:3:p:197-229

Access Statistics for this article

More articles in INFORMS Joural on Data Science from INFORMS Contact information at EDIRC.
Bibliographic data for series maintained by Chris Asher ().