Is UWLS Really Better? A Replication and Pre-Registered Robustness Check of Stanley et al., Journal of Clinical Epidemiology (2023)

Sanghyun Hong () and W. Reed ()
Additional contact information
Sanghyun Hong: University of Canterbury, https://www.canterbury.ac.nz

Working Papers in Economics from University of Canterbury, Department of Economics and Finance

Abstract: This study critically examines the reproducibility and robustness of Stanley et al. (2023). Stanley et al. analyzed 67,308 meta-analysis datasets from the Cochrane Database of Systematic Reviews (CDSR). They find that a generalized Fixed Effect estimator they call unrestricted weighted least squares (UWLS-FE) is determined by the model selection criteria AIC and BIC to be superior to Random Effects (RE). In some cases, it is also better than Fixed Effects. Our analysis uses their original data and code and exactly reproduces their results. However, we show that the small sample sizes of the CDSR datasets undermine the reliability of AIC and BIC for model selection. Accordingly, we simulate 108,000 datasets mirroring the original CDSR data. This allows us to know the true model parameters and evaluate the estimators more accurately. Our findings suggest that RE generally outperforms UWLS-FE. The comparison with FE is less clear. FE frequently produces more accurate standard errors than UWLS-FE, making confidence intervals and hypothesis testing more reliable. However, for some types of data, UWLS-FE may be better than FE.

Keywords: Meta-analysis; Unrestricted Weighted Least Squares; Fixed Effect; Random Effects; Medical Research; Cochrane Database of Systematic Reviews; Replication; Robustness Check; Pre-Registration (search for similar items in EconPapers)
JEL-codes: B4 C18 I1 (search for similar items in EconPapers)
Pages: 30 pages
Date: 2025-01-01
References: View complete reference list from CitEc
Citations:

Downloads: (external link)
https://repec.canterbury.ac.nz/cbt/econwp/2501.pdf (application/pdf)

Related works:
Working Paper: Is UWLS Really Better? A Replication and Pre-Registered Robustness Check of Stanley et al., Journal of Clinical Epidemiology (2023) (2024)
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:cbt:econwp:25/01

Access Statistics for this paper

More papers in Working Papers in Economics from University of Canterbury, Department of Economics and Finance Private Bag 4800, Christchurch, New Zealand. Contact information at EDIRC.
Bibliographic data for series maintained by Albert Yee ().