An Empirical Validation of the Regression Point Displacement Design Using Within-Study Comparison Logic: Emerging Possibilities and Cautions

Joshua Hendrickse and William H. Yeaton

Evaluation Review, 2021, vol. 45, issue 6, 279-308

Abstract: Background The regression point displacement (RPD) design is a quasi-experiment (QE) that aims to control many threats to internal validity. Though it has existed for several decades, RPD has only recently begun to answer applied research questions in lieu of stronger QEs. Objectives Our primary objective was to implement within-study comparison (WSC) logic to create RPD replicates and to determine conditions under which RPD might provide estimates comparable to those found in validating experiments. Research Design We utilize three randomized controlled trials (two cluster-level, one individual-level), artificially decomposing or creating cluster structures, to create multiple RPDs. We compare results in each RPD treatment group to a fixed set of control groups to gauge the congruence of these repeated RPD realizations with results found in these three RCTs. Results RPD’s performance was uneven. Using multiple criteria, we found that RPDs successfully predicted the direction of the RCT’s intervention effect but inconsistently fell within the .10 SD threshold. A scant 13% of RPD results were statistically significant at either the .05 or .01 alpha-level. RPD results were within the 95% confidence interval of RCTs around half the time, and false negative rates were substantially higher than false positive rates. Conclusions RPD consistently underestimates treatment effects in validating RCTs. We analyze reasons for this insensitivity and offer practical suggestions to improve the chances RPD will correctly identify favorable results. We note that the synthetic, “decomposition of cluster RCTs,†WSC design represents a prototype for evaluating other QEs.

Keywords: regression point displacement; validation; within-study comparison; cluster randomized controlled trial; decomposition of cluster randomized controlled trials (search for similar items in EconPapers)
Date: 2021
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
https://journals.sagepub.com/doi/10.1177/0193841X211064420 (text/html)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:sae:evarev:v:45:y:2021:i:6:p:279-308

DOI: 10.1177/0193841X211064420

Access Statistics for this article

More articles in Evaluation Review
Bibliographic data for series maintained by SAGE Publications ().