Sensitivity analysis for p-hacking in meta-analyses

Maya B Mathur

No ezjsx, OSF Preprints from Center for Open Science

Abstract: As traditionally conceived, publication bias arises from selection operating on a collection of individually unbiased estimates. A canonical form of such selection across studies (SAS) is the preferential publication of affirmative studies (i.e., significant, positive estimates) versus nonaffirmative studies (i.e., nonsignificant or negative estimates). However, meta-analyses can also be compromised by selection within studies (SWS), in which investigators “p-hack’’ results within their study to obtain an affirmative estimate. Published estimates can then be biased even conditional on affirmative status, compromising existing methods that only consider SAS. We propose two sensitivity analyses that accommodate joint SAS and SWS; both analyze only the published nonaffirmative estimates. First, assuming that published, hacked studies never have nonaffirmative estimates (e.g., their investigators p-hack until they obtain an affirmative estimate), we propose estimating the underlying meta-analytic mean by fitting “right-truncated meta-analysis’’ (RTMA) to the published nonaffirmative estimates, which are unhacked. Second, we propose conducting a standard meta-analysis of only the nonaffirmative studies (MAN); this estimate is conservative (negatively biased) under weakened assumptions, including when nonaffirmative estimates from p-hacked studies are sometimes published. We provide an R package, phacking. Our proposed methods supplement existing methods by assessing the robustness of meta-analyses to joint SAS and SWS.

Date: 2022-06-01
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
https://osf.io/download/62977f0f5ae61005cdadd9ee/

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:osf:osfxxx:ezjsx

DOI: 10.31219/osf.io/ezjsx

Access Statistics for this paper

More papers in OSF Preprints from Center for Open Science
Bibliographic data for series maintained by OSF ().