Probability-based A/B testing with Adaptive Minimax Regret (AMR) criterion for long-term customer metrics

Makoto Abe
Additional contact information
Makoto Abe: Faculty of Economics, The University of Tokyo

No CIRJE-J-313, CIRJE J-Series from CIRJE, Faculty of Economics, University of Tokyo

Abstract: In consumer research and psychological experiments, subjects' states (attitudes) are manipulated by means of stimulus treatment in order to examine the effects of the subjects' states (attitudes) on the target variable. The interest here is not the effect of the treatment (stimulus) itself, but the effect on the target variable of the difference in state produced as a result of the treatment. Therefore, a manipulation check is usually performed to establish the validity of the experimental design, i.e., whether the stimulus produced the intended difference in state. When the manipulation-check variable (state) is directly associated with the target variable, one encounters the problem of confounding that affects both variables. To eliminate this problem, randomized controlled trials (RCTs) are used, but two weaknesses exist: first, only a discrete, binary effect of the presence or absence of an treatment on the target variable can be uncovered. Second, the incompleteness of the experimental design, in which the state induced by the treatment (stimulus) varies from subject to subject, resulting in different effects on the target variable, cannot be taken into account. In this study, we propose an approach that can correctly estimate the effect, which relates the manipulation-check variable to the target variable, even when unobserved confounding factors are present. By accounting for imperfections in the experimental design, the effect of the state variable becomes statistically more efficient than the effect of the experimental approach. The simulation analysis confirms that, for the same sample size, our instrumental variable approach is more significant than the usual experimental approach.

Pages: 15 pages
Date: 2025-03
New Economics Papers: this item is included in nep-ecm and nep-exp
References: Add references at CitEc
Citations:

Downloads: (external link)
http://www.cirje.e.u-tokyo.ac.jp/research/dp/2025/2025cj312.pdf (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:tky:jseres:2025cj312

Access Statistics for this paper

More papers in CIRJE J-Series from CIRJE, Faculty of Economics, University of Tokyo Contact information at EDIRC.
Bibliographic data for series maintained by CIRJE administrative office (cirje@e.u-tokyo.ac.jp).