Standardized effect sizes are far from “Standardized”: A primer and empirical illustration in depression psychotherapy meta-analyses

Mathias Harrer, Clara Miguel, Yan Luo, Edoardo G Ostinelli, Eirini Karyotaki, Stefan Leucht, Toshi A Furukawa and Pim Cuijpers

PLOS Mental Health, 2025, vol. 2, issue 7, 1-16

Abstract: Standardized mean differences (SMDs) are frequently used to appraise the effects of psychological treatments, and to combine them in meta-analyses. Yet, there is no consensus on how exactly SMDs should be computed from randomized trials. In this study, we show that different SMD variants can heavily diverge in aggregate-data meta-analyses, subverting the original purpose of standardization. We investigate the impact this has on the estimated benefits of depression psychotherapies. Different SMD versions using endpoint or change scores were calculated from a comprehensive database of randomized trials, comparing depression psychotherapy against pharmacotherapy and inactive controls. Pooled treatment effects were obtained for each variant, assuming correlations between baseline and endpoint scores of 0.2 through 0.8, and their relationship was examined using bivariate meta-analyses. We also investigated which study characteristics predicted divergent effect estimates. A total of k = 443 trials with 48,221 participants were analyzed. The pooled effect of psychotherapy versus controls varied heavily depending on the calculation methods (SMD = 0.65–1.24), even though the same studies were used. Divergences were less pronounced for psychotherapies compared to pharmacotherapy (SMD = 0.05–0.14). Change score SMDs deviated from endpoint SMDs especially when high (r = 0.8) or low (r = 0.2) pre-post correlations were assumed. This difference was largest in subfields with high treatment effects. Different SMD calculation methods can lead to strongly diverging effect estimates of psychological treatment; especially when change scores are used and pre-post correlations are very high or low. This could have a profound impact on how treatment benefits are interpreted within and across meta-analyses. Researchers could prioritize endpoint SMDs of randomized trials, and should consider standardization using population-level estimates to improve the comparability of meta-analytic effects in the field.Open Material; Registration: htpps://doi.org/10.5281/zenodo.10694719; https://osf.io/yx5jg; https://osf.io/4j23t.

Date: 2025
References: Add references at CitEc
Citations:

Downloads: (external link)
https://journals.plos.org/mentalhealth/article?id=10.1371/journal.pmen.0000347 (text/html)
https://journals.plos.org/mentalhealth/article/fil ... 00347&type=printable (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:plo:pmen00:0000347

DOI: 10.1371/journal.pmen.0000347

Access Statistics for this article

More articles in PLOS Mental Health from Public Library of Science
Bibliographic data for series maintained by mentalhealth ().