Efficiently Evaluating Targeting Policies: Improving on Champion vs. Challenger Experiments

Duncan Simester (), Artem Timoshenko () and Spyros I. Zoumpoulis ()
Additional contact information
Duncan Simester: Massachusetts Institute of Technology, Cambridge, Massachusetts 02139
Artem Timoshenko: Massachusetts Institute of Technology, Cambridge, Massachusetts 02139
Spyros I. Zoumpoulis: INSEAD, 77300 Fontainebleau, France

Management Science, 2020, vol. 66, issue 8, 3412-3424

Abstract: Champion versus challenger field experiments are widely used to compare the performance of different targeting policies. These experiments randomly assign customers to receive marketing actions recommended by either the existing (champion) policy or the new (challenger) policy, and then compare the aggregate outcomes. We recommend an alternative experimental design and propose an alternative estimation approach to improve the evaluation of targeting policies. The recommended experimental design randomly assigns customers to marketing actions. This allows evaluation of any targeting policy without requiring an additional experiment, including policies designed after the experiment is implemented. The proposed estimation approach identifies customers for whom different policies recommend the same action and recognizes that for these customers there is no difference in performance. This allows for a more precise comparison of the policies. We illustrate the advantages of the experimental design and estimation approach using data from an actual field experiment. We also demonstrate that the grouping of customers, which is the foundation of our estimation approach, can help to improve the training of new targeting policies.

Keywords: targeting; field experiments; machine learning; counterfactual policy logging; policy evaluation (search for similar items in EconPapers)
Date: 2020
References: View references in EconPapers View complete reference list from CitEc
Citations: View citations in EconPapers (8)

Downloads: (external link)
https://doi.org/10.1287/mnsc.2019.3379 (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:inm:ormnsc:v:66:y:2020:i:8:p:3412-3424

Access Statistics for this article

More articles in Management Science from INFORMS Contact information at EDIRC.
Bibliographic data for series maintained by Chris Asher ().