INTERNAL AND EXTERNAL VALIDITY OF THE COMPARATIVE INTERRUPTED TIME‐SERIES DESIGN: A META‐ANALYSIS

Jared Coopersmith, Thomas D. Cook, Jelena Zurovac, Duncan Chaplin and Lauren V. Forrow

Journal of Policy Analysis and Management, 2022, vol. 41, issue 1, 252-277

Abstract: This paper meta‐analyzes 12 heterogeneous studies that examine bias in the comparative interrupted time‐series design (CITS) that is often used to evaluate the effects of social policy interventions. To measure bias, each CITS impact estimate was differenced from the estimate derived from a theoretically unbiased causal benchmark study that tested the same hypothesis with the same treatment group, outcome data, and estimand. In 10 studies, the benchmark was a randomized experiment and in the other two it was a regression‐discontinuity study. Analyses revealed the average standardized CITS bias to be between −0.01 and 0.042 standard deviations; and all but one bias estimate from individual studies fell within 0.10 standard deviations of its benchmark, indicating that the near zero mean bias did not result from averaging many large single study differences. The low mean and generally tight distribution of individual bias estimates suggest that CITS studies are worth recommending for future causal hypothesis tests because: (1) over the studies examined, they generally resulted in high internal validity; and (2) they also promise high external validity because the empirical tests we synthesized occurred across a wide variety of settings, times, interventions, and outcomes.

Date: 2022
References: View references in EconPapers View complete reference list from CitEc
Citations: View citations in EconPapers (1)

Downloads: (external link)
https://doi.org/10.1002/pam.22361

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:wly:jpamgt:v:41:y:2022:i:1:p:252-277

Access Statistics for this article

More articles in Journal of Policy Analysis and Management from John Wiley & Sons, Ltd.
Bibliographic data for series maintained by Wiley Content Delivery ().