Using Monte Carlo Experiments to Select Meta-Analytic Estimators

Sanghyun Hong () and W. Reed ()
Additional contact information
Sanghyun Hong: University of Canterbury, https://www.canterbury.ac.nz

Working Papers in Economics from University of Canterbury, Department of Economics and Finance

Abstract: The purpose of this study is to show how Monte Carlo analysis of meta-analytic estimators can be used to select estimators for specific research situations. Our analysis conducts 1,620 individual experiments, where each experiment is defined by a unique combination of sample size, effect heterogeneity, effect size, publication selection mechanism, and other research characteristics. We compare eleven estimators commonly used in medicine, psychology, and the social sciences. These are evaluated on the basis of bias, mean squared error (MSE), and coverage rates. For our experimental design, we reproduce simulation environments from four recent studies: Stanley, Doucouliagos, & Ioannidis (2017), Alinaghi & Reed (2018), Bom & Rachinger (2019), and Carter et al. (2019a). We demonstrate that relative estimator performance differs across performance measures. An estimator that may be especially good with respect to MSE may perform relatively poorly with respect to coverage rates. We also show that sample size and effect heterogeneity are important determinants of relative estimator performance. We use these results to demonstrate how the observable characteristics of sample size and effect heterogeneity can guide the meta-analyst in choosing the estimators most appropriate for their research circumstances. All of the programming code and output files associated with this project are available at https://osf.io/pr4mb/.

Keywords: Meta-analysis; Estimator performance; Publication bias; Simulation design; Monte Carlo; Experiments (search for similar items in EconPapers)
JEL-codes: B41 C15 C18 (search for similar items in EconPapers)
Pages: 46 pages
Date: 2020-06-01
New Economics Papers: this item is included in nep-ecm, nep-exp and nep-ore
References: View references in EconPapers View complete reference list from CitEc
Citations: View citations in EconPapers (5)

Downloads: (external link)
https://repec.canterbury.ac.nz/cbt/econwp/2010.pdf (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:cbt:econwp:20/10

Access Statistics for this paper

More papers in Working Papers in Economics from University of Canterbury, Department of Economics and Finance Private Bag 4800, Christchurch, New Zealand. Contact information at EDIRC.
Bibliographic data for series maintained by Albert Yee ().