Counting Defiers: A Design-Based Model of an Experiment Can Reveal Evidence Beyond the Average Effect

Neil Christy and Amanda Kowalski

Papers from arXiv.org

Abstract: Leveraging structure from the randomization process, a design-based model of an experiment with a binary intervention and outcome can reveal evidence beyond the average effect without additional data. Our proposed statistical decision rule yields a design-based maximum likelihood estimate (MLE) of the joint distribution of potential outcomes in intervention and control, specified by the numbers of always takers, compliers, defiers, and never takers in the sample. With a visualization, we explain why the likelihood varies with the number of defiers within the Frechet bounds determined by the estimated marginal distributions. We illustrate how the MLE varies with all possible data in samples of 50 and 200: when the estimated average effect is positive, the MLE includes defiers if takeup is below half in control and above half in intervention, unless takeup is zero in control or full in intervention. Under optimality conditions, for increasing sample sizes in which exhaustive grid search is possible, our rule's performance increases relative to a rule that places equal probability on all numbers of defiers within the estimated Frechet bounds. We offer insights into effect heterogeneity in two published experiments with positive, statistically significant average effects on takeup of desired health behaviors and plausible defiers. Our 95% smallest credible sets for defiers include zero and the estimated upper Frechet bound, demonstrating that evidence is weak. Yet, our rule yields no defiers in one experiment. In the other, our rule yields the estimated upper Frechet bound on defiers -- a count representing over 18% of the sample.

Date: 2024-12, Revised 2025-10
New Economics Papers: this item is included in nep-ecm, nep-exp, nep-hea and nep-upt
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
http://arxiv.org/pdf/2412.16352 Latest version (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:arx:papers:2412.16352

Access Statistics for this paper

More papers in Papers from arXiv.org
Bibliographic data for series maintained by arXiv administrators ().