Design Parameters for Planning the Sample Size of Individual-Level Randomized Controlled Trials in Community Colleges

Marie-Andrée Somers, Michael J. Weiss and Colin Hill

Evaluation Review, 2023, vol. 47, issue 4, 599-629

Abstract: The last two decades have seen a dramatic increase in randomized controlled trials (RCTs) conducted in community colleges. Yet, there is limited empirical information on the design parameters necessary to plan the sample size for RCTs in this context. For a blocked student-level random assignment research design, key design parameters for the minimum detectable true effect (MDTE) are the within-block outcome standard deviation ( σ | S ) and the within-block outcome variance explained by baseline covariates like student characteristics ( R | S 2 ) . We provide empirical estimates of these key design parameters, discussing the pattern of estimates across outcomes (enrollment, credits earned, credential attainment, and grade point average), semesters, and studies. The main analyses use student-level data from 8 to 14 RCTs including 5,649–7,099 students (depending on the outcome) with follow-up data for 3 years. The following patterns are observed: the within-block standard deviation ( σ | S ) and therefore the MDTE can be much larger in later semesters for enrollment outcomes and cumulative credits earned; there is substantial variation across studies in σ | S for degree attainment; and baseline covariates explain less than 10% of the variation in student outcomes. These findings indicate that when planning the sample size for a study, researchers should be mindful of the follow-up period, use a range of values to calculate the MDTE for outcomes that vary across studies, and assume a value of R | S 2 between 0 and 0.05. A public database created for this paper includes parameter estimates for additional RCTs and students.

Keywords: design parameters; statistical power; community college; postsecondary; randomized controlled trial (search for similar items in EconPapers)
Date: 2023
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
https://journals.sagepub.com/doi/10.1177/0193841X221121236 (text/html)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:sae:evarev:v:47:y:2023:i:4:p:599-629

DOI: 10.1177/0193841X221121236

Access Statistics for this article

More articles in Evaluation Review
Bibliographic data for series maintained by SAGE Publications ().