Learning from Many Experiments: A Hierarchical Bayesian Framework for Decomposing Marketing Treatment Heterogeneity

Peter Ebbes, Eva Ascarza and Oded Netzer
Additional contact information
Peter Ebbes: HEC Paris - Ecole des Hautes Etudes Commerciales
Eva Ascarza: Havard Business School
Oded Netzer: Columbia University [New York]

Working Papers from HAL

Abstract: Firms increasingly rely on A/B testing to evaluate marketing strategies, yet most experiments are analyzed in isolation, limiting insight into why effectiveness varies and how repeated exposure shapes outcomes. We develop a hierarchical Bayesian framework that jointly analyzes randomized marketing interventions to decompose treatment effect heterogeneity into three components: customer responsiveness, campaign design, and contextual timing. The model is scalable, incorporates both observed and unobserved variation, and links estimation directly to policy evaluation with full uncertainty quantification. Using data from large-scale field experiments involving nearly half a million customers, we document three main findings. First, unobserved customer-level heterogeneity accounts for the majority of variation in treatment effects, explaining roughly two-thirds of the total dispersion, considerably exceeding the contribution of campaign design or timing. Second, we find strong evidence of intervention fatigue: responsiveness declines with repeated exposure and recovers only slowly. Third, in a held-out policy evaluation, a model-based targeting strategy improves revenue on average while substantially reducing the share of customers targeted, with gains reaching over 10% in settings where experimental variation is most informative. Approximately 80% of these gains are driven by personalization based on latent customer responsiveness rather than observable characteristics. Together, these results demonstrate how repeated experimentation can be leveraged to explain variation in marketing effectiveness and to support scalable targeting and personalization.

Keywords: Marketing interventions; heterogeneity; field experiments; Bayesian modeling; treatment effect decomposition; targeting (search for similar items in EconPapers)
Date: 2026-03-23
References: Add references at CitEc
Citations:

There are no downloads for this item, see the EconPapers FAQ for hints about obtaining it.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:hal:wpaper:hal-05562965

DOI: 10.2139/ssrn.6213079

Access Statistics for this paper

More papers in Working Papers from HAL
Bibliographic data for series maintained by CCSD ().