Using Extant Data to Improve Estimation of the Standardized Mean Difference

Kaitlyn G. Fitzgerald and Elizabeth Tipton
Additional contact information
Kaitlyn G. Fitzgerald: Azusa Pacific University
Elizabeth Tipton: Northwestern University

Journal of Educational and Behavioral Statistics, 2025, vol. 50, issue 1, 128-148

Abstract: This article presents methods for using extant data to improve the properties of estimators of the standardized mean difference (SMD) effect size. Because samples recruited into education research studies are often more homogeneous than the populations of policy interest, the variation in educational outcomes can be smaller in these samples than is reflective of the true variation in the population. This affects effect size estimation since the sample standard deviation is used in the denominator of the SMD. We propose leveraging extant data on sample variance estimates from multiple studies, made available via clearinghouse databases such as the What Works Clearinghouse, to standardize a mean difference. This allows effect sizes to be benchmarked across a common and broad population, thus enabling better comparability across studies and interventions. We derive the new estimators of the population variance and the corresponding SMD, which pool sample variances from multiple studies using both an analysis of variance and a meta-analytic framework. We demonstrate the properties of these estimators via analytic and simulation results and offer recommendations for when these estimators are appropriate in practice.

Keywords: effect size; meta-analysis; ANOVA; secondary data analysis; research utilization; educational policy (search for similar items in EconPapers)
Date: 2025
References: Add references at CitEc
Citations:

Downloads: (external link)
https://journals.sagepub.com/doi/10.3102/10769986241238478 (text/html)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:sae:jedbes:v:50:y:2025:i:1:p:128-148

DOI: 10.3102/10769986241238478

Access Statistics for this article

More articles in Journal of Educational and Behavioral Statistics
Bibliographic data for series maintained by SAGE Publications ().