Evaluating the Validity of Post-Hoc Subgroup Inferences: A Case Study

Joseph J. Lee and Donald B. Rubin

The American Statistician, 2016, vol. 70, issue 1, 39-46

Abstract: In randomized experiments, the random assignment of units to treatment groups justifies many of the widely used traditional analysis methods for evaluating causal effects. Specifying subgroups of units for further examination after observing outcomes, however, may partially nullify any advantages of randomized assignment when data are analyzed naively. Some previous statistical literature has treated all post-hoc subgroup analyses homogeneously as entirely invalid and thus uninterpretable. The extent of the validity of such analyses and the factors that affect the degree of validity remain largely unstudied. Here, we describe a recent pharmaceutical case with First Amendment legal implications, in which post-hoc subgroup analyses played a pivotal and controversial role. Through Monte Carlo simulation, we show that post-hoc results that seem highly significant make dramatic movements toward insignificance after accounting for the subgrouping procedure presumably used. Finally, we propose a novel, randomization-based method that generates valid post-hoc subgroup p -values, provided we know exactly how the subgroups were constructed. If we do not know the exact subgrouping procedure, our method may still place helpful bounds on the significance level of estimated effects. This randomization-based approach allows us to evaluate causal effects in situations where valid evaluations were previously considered impossible.[Received February 2014. Revised April 2015.]

Date: 2016
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
http://hdl.handle.net/10.1080/00031305.2015.1093961 (text/html)
Access to full text is restricted to subscribers.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:taf:amstat:v:70:y:2016:i:1:p:39-46

Ordering information: This journal article can be ordered from
http://www.tandfonline.com/pricing/journal/UTAS20

DOI: 10.1080/00031305.2015.1093961

Access Statistics for this article

The American Statistician is currently edited by Eric Sampson

More articles in The American Statistician from Taylor & Francis Journals
Bibliographic data for series maintained by Chris Longhurst ().