A Nonparametric Bayesian Analysis of Heterogenous Treatment Effects in Digital Experimentation

Matt Taddy, Matt Gardner, Liyun Chen and David Draper

Journal of Business & Economic Statistics, 2016, vol. 34, issue 4, 661-672

Abstract: Randomized controlled trials play an important role in how Internet companies predict the impact of policy decisions and product changes. In these “digital experiments,” different units (people, devices, products) respond differently to the treatment. This article presents a fast and scalable Bayesian nonparametric analysis of such heterogenous treatment effects and their measurement in relation to observable covariates. New results and algorithms are provided for quantifying the uncertainty associated with treatment effect measurement via both linear projections and nonlinear regression trees (CART and random forests). For linear projections, our inference strategy leads to results that are mostly in agreement with those from the frequentist literature. We find that linear regression adjustment of treatment effect averages (i.e., post-stratification) can provide some variance reduction, but that this reduction will be vanishingly small in the low-signal and large-sample setting of digital experiments. For regression trees, we provide uncertainty quantification for the machine learning algorithms that are commonly applied in tree-fitting. We argue that practitioners should look to ensembles of trees (forests) rather than individual trees in their analysis. The ideas are applied on and illustrated through an example experiment involving 21 million unique users of EBay.com.

Date: 2016
References: View references in EconPapers View complete reference list from CitEc
Citations: View citations in EconPapers (31)

Downloads: (external link)
http://hdl.handle.net/10.1080/07350015.2016.1172013 (text/html)
Access to full text is restricted to subscribers.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:taf:jnlbes:v:34:y:2016:i:4:p:661-672

Ordering information: This journal article can be ordered from
http://www.tandfonline.com/pricing/journal/UBES20

DOI: 10.1080/07350015.2016.1172013

Access Statistics for this article

Journal of Business & Economic Statistics is currently edited by Eric Sampson, Rong Chen and Shakeeb Khan

More articles in Journal of Business & Economic Statistics from Taylor & Francis Journals
Bibliographic data for series maintained by Chris Longhurst ().