Towards an Experimental Framework for Assessing Meta-Analysis Methods, with a Focus on Andrews-Kasy Estimators

Sanghyun Hong () and W. Reed ()
Additional contact information
Sanghyun Hong: University of Canterbury, https://www.canterbury.ac.nz

Working Papers in Economics from University of Canterbury, Department of Economics and Finance

Abstract: The study contributes towards the development of a systematic experimental framework for evaluating meta-analysis methods. Towards that goal, we reproduce the Monte Carlo experiments from three studies: Stanley & Doucouliagos (2017); Stanley, Doucouliagos, & Ioannidis (2017); and Alinaghi & Reed (2018) – S&D, SD&I, and A&R, respectively. We demonstrate that the relative performance of estimators depends on whether the researcher is concerned with unbiasedness, mean squared error (MSE), or coverage rates. We also show how estimator performance varies systematically with the number of estimates in the meta-analyst’s sample and the degree of effect heterogeneity as measured by I2. This demonstrates the possibility that researchers can select a “best estimator” based on the observable characteristics of their meta-analysis samples. We further show that the design of simulation experiments makes a difference: Different simulation designs by S&D and SD&I applied to the same “types” of meta-analysis samples select different “best” estimators. Different designs in A&R also produce different results. This highlights the need to know more about which aspects of simulation designs are important for estimator performance. Finally, our results indicate that the recent Andrews & Kasy (2019) estimators perform well in a number of research environments, frequently outperforming the popular PET-PEESE and WAAP estimators, though more research is needed.

Keywords: Meta-analysis; Estimator performance; Publication bias; Simulation design; WAAP; PET-PEESE; Andrews-Kasy; Monte Carlo; Experiments (search for similar items in EconPapers)
JEL-codes: B41 C15 C18 (search for similar items in EconPapers)
Pages: 56 pages
Date: 2019-10-01
New Economics Papers: this item is included in nep-ecm, nep-exp and nep-ore
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
https://repec.canterbury.ac.nz/cbt/econwp/1913.pdf (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:cbt:econwp:19/13

Access Statistics for this paper

More papers in Working Papers in Economics from University of Canterbury, Department of Economics and Finance Private Bag 4800, Christchurch, New Zealand. Contact information at EDIRC.
Bibliographic data for series maintained by Albert Yee ().