Is UWLS Really Better? A Replication and Pre-Registered Robustness Check of Stanley et al., Journal of Clinical Epidemiology (2023)

Sanghyun Hong () and W. Reed ()
Additional contact information
Sanghyun Hong: University of Canterbury, https://www.canterbury.ac.nz

Working Papers in Economics from University of Canterbury, Department of Economics and Finance

Abstract: This study investigates the reproducibility and robustness of Stanley et al. (2023). Stanley et al. (2023) studied 67,308 meta-analyses of medical research from the Cochrane Database of Systematic Reviews (CDSR). They compared estimators using two information criteria: the Bayes Information Criterion (BIC) and the Akaike Information Criterion (AIC). They concluded that a variant of the Fixed Effect (FE) estimator they call Unrestricted Weighted Least Squares (UWLS) is “a better model of medical research than Random Effects (RE) regardless of heterogeneity, number of studies, or the type of outcome.” With respect to reproducibility, we can report that we are able to exactly reproduce Stanley et al. (2023)’s results. To test the robustness of their findings, we simulated 108,000 meta-analyses designed to represent the datasets in Stanley et al. (2023). This allowed us to assess estimator performance using bias, mean-squared error, and coverage rates. The overall conclusion we draw from our analysis is that BIC and AIC do not reliably guide the researcher to the “best” estimator when estimating mean treatment effects for medical research in Stanley et al. (2023)’s dataset. Further, the dominance of UWLS as measured by BIC and AIC is not matched by similar dominance on bias, MSE, and coverage rates. Most importantly, there is no evidence to support the headline result from Stanley et al. (2023) that UWLS dominates RE.

Keywords: Meta-analysis; Unrestricted Weighted Least Squares; Fixed Effect; Random Effects; Medical Research; Cochrane Database of Systematic Reviews; Replication; Robustness Check; Pre-Registration (search for similar items in EconPapers)
JEL-codes: B4 C18 I1 (search for similar items in EconPapers)
Pages: 32 pages
Date: 2024-05-01
New Economics Papers: this item is included in nep-inv
References: View complete reference list from CitEc
Citations:

Downloads: (external link)
https://repec.canterbury.ac.nz/cbt/econwp/2407.pdf (application/pdf)

Related works:
Working Paper: Is UWLS Really Better? A Replication and Pre-Registered Robustness Check of Stanley et al., Journal of Clinical Epidemiology (2023) (2025)
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:cbt:econwp:24/07

Access Statistics for this paper

More papers in Working Papers in Economics from University of Canterbury, Department of Economics and Finance Private Bag 4800, Christchurch, New Zealand. Contact information at EDIRC.
Bibliographic data for series maintained by Albert Yee ().