A Performance Analysis of Some New Meta-Analysis Estimators Designed to Correct Publication Bias

Sanghyun Hong () and W. Reed ()
Additional contact information
Sanghyun Hong: University of Canterbury, https://www.canterbury.ac.nz

Working Papers in Economics from University of Canterbury, Department of Economics and Finance

Abstract: Publication selection bias is widely recognized as a serious challenge to the validity of meta-analyses. This study analyses the performance of three new estimators designed to correct publication bias: the weighted average of the adequately powered (WAAP) estimator of Stanley et al. (2017), and two estimators proposed by Andrews & Kasy (2019), which we call AK1 and AK2. With respect to bias, we find that none of these is consistently superior to the commonly used PET-PEESE estimator. With respect to mean squared error, we find that Andrews & Kasey’s AK1 estimator does consistently better than other estimators except when publication bias is focused solely on the sign, as opposed to the significance, of an effect. With respect to coverage rates, we find that all the estimators perform consistently poorly, so that hypothesis tests about the mean true effect are unreliable. We also find that effect heterogeneity generally worsens estimator performance, and that its adverse impact compounds with greater heterogeneity. This is particularly of concern for meta-analyses in business and economics, where I2 values, a measure of heterogeneity, are often 90 percent or higher. Finally, we find that the type of simulation environment used in the Monte Carlo experiments significantly impacts estimator performance. A better understanding of what makes an “appropriate” simulation environment for analysing meta-analysis estimators would be a potentially productive subject for future research.

Keywords: Meta-analysis; publication bias; WAAP; Andrews-Kasy; Monte Carlo; Simulations (search for similar items in EconPapers)
JEL-codes: B41 C15 C18 (search for similar items in EconPapers)
Pages: 58 pages
Date: 2019-04-01
New Economics Papers: this item is included in nep-ecm and nep-sog
References: View references in EconPapers View complete reference list from CitEc
Citations: View citations in EconPapers (3)

Downloads: (external link)
https://repec.canterbury.ac.nz/cbt/econwp/1904.pdf (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:cbt:econwp:19/04

Access Statistics for this paper

More papers in Working Papers in Economics from University of Canterbury, Department of Economics and Finance Private Bag 4800, Christchurch, New Zealand. Contact information at EDIRC.
Bibliographic data for series maintained by Albert Yee ().