Applications of a Within-Study Comparison Approach for Evaluating Bias in Generalized Causal Inferences From Comparison Groups Studies

Andrew P. Jaciw

Evaluation Review, 2016, vol. 40, issue 3, 241-276

Abstract: Background: Past studies have examined factors associated with reductions in bias in comparison group studies (CGSs). The companion work to this article extends the framework to investigate the accuracy of generalized inferences from CGS. Objectives: This article empirically examines levels of bias in CGS-based impact estimates when used for generalization, and reductions in bias resulting from covariate adjustment. It assesses potential for bias reduction against criteria from past studies. Research design: Multisite trials are used to generate impact estimates based on cross-site comparisons that are evaluated against site-specific experimental benchmarks. Strategies for reducing bias are evaluated. Results from two experiments are considered. Subjects: Students in Grades K–3 in 79 schools in Tennessee and students in Grades 4–8 in 82 schools in Alabama. Measures: Grades K–3 Stanford Achievement Test reading and math scores; Grades 4–8 Stanford Achievement Test (SAT) 10 reading scores. Results: Generalizing impacts to sites through estimates based on between-site nonexperimental comparisons leads to bias from differences between sites in average performance, and in impact, and covariation between these quantities. The first of these biases is larger. Covariate adjustments reduce bias but not completely. Criteria for bias reduction from past studies appear to extend to generalized inferences based on CGSs. Conclusion: When generalizing from a CGS, results may be affected by bias from differences between the study and inference sites in both average performance and average impact. The same factors may underlie both forms of bias. Researchers and practitioners can assess the validity of generalized inferences from CGSs by applying criteria for bias reduction from past studies.

Keywords: bias in quasi-experiments; within-study comparison approaches; comparison group studies (search for similar items in EconPapers)
Date: 2016
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
https://journals.sagepub.com/doi/10.1177/0193841X16664457 (text/html)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:sae:evarev:v:40:y:2016:i:3:p:241-276

DOI: 10.1177/0193841X16664457

Access Statistics for this article

More articles in Evaluation Review
Bibliographic data for series maintained by SAGE Publications ().