An Approach for Addressing the Multiple Testing Problem in Social Policy Impact Evaluations
Peter Z. Schochet
Additional contact information
Peter Z. Schochet: Mathematica Policy Research, Inc., Princeton, New Jersey, email@example.com
Evaluation Review, 2009, vol. 33, issue 6, 539-567
In social policy evaluations, the multiple testing problem occurs due to the many hypothesis tests that are typically conducted across multiple outcomes and subgroups, which can lead to spurious impact findings. This article discusses a framework for addressing this problem that balances Types I and II errors. The framework involves specifying confirmatory and exploratory analyses in study protocols, delineating confirmatory outcome domains, conducting t tests on composite domain outcomes, and applying multiplicity corrections to composites across domains to obtain summative impact evidence. The article presents statistical background and discusses multiplicity issues for subgroup analyses, designs with multiple treatments, and reporting.
Keywords: multiple comparisons; impact evaluations; design of experiments (search for similar items in EconPapers)
References: Add references at CitEc
Citations View citations in EconPapers (23) Track citations by RSS feed
Downloads: (external link)
This item may be available elsewhere in EconPapers: Search for items with the same title.
Export reference: BibTeX
RIS (EndNote, ProCite, RefMan)
Persistent link: https://EconPapers.repec.org/RePEc:sae:evarev:v:33:y:2009:i:6:p:539-567
Access Statistics for this article
More articles in Evaluation Review
Bibliographic data for series maintained by SAGE Publications ().