An Approach for Addressing the Multiple Testing Problem in Social Policy Impact Evaluations

Peter Z. Schochet
Additional contact information
Peter Z. Schochet: Mathematica Policy Research, Inc., Princeton, New Jersey, pschochet@mathematica-mpr.com

Evaluation Review, 2009, vol. 33, issue 6, 539-567

Abstract: In social policy evaluations, the multiple testing problem occurs due to the many hypothesis tests that are typically conducted across multiple outcomes and subgroups, which can lead to spurious impact findings. This article discusses a framework for addressing this problem that balances Types I and II errors. The framework involves specifying confirmatory and exploratory analyses in study protocols, delineating confirmatory outcome domains, conducting t tests on composite domain outcomes, and applying multiplicity corrections to composites across domains to obtain summative impact evidence. The article presents statistical background and discusses multiplicity issues for subgroup analyses, designs with multiple treatments, and reporting.

Keywords: multiple comparisons; impact evaluations; design of experiments (search for similar items in EconPapers)
Date: 2009
References: View references in EconPapers View complete reference list from CitEc
Citations: View citations in EconPapers (31)

Downloads: (external link)
https://journals.sagepub.com/doi/10.1177/0193841X09350590 (text/html)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:sae:evarev:v:33:y:2009:i:6:p:539-567

DOI: 10.1177/0193841X09350590

Access Statistics for this article

More articles in Evaluation Review
Bibliographic data for series maintained by SAGE Publications ().