Meta-regression to explain shrinkage and heterogeneity in large-scale replication projects

Rachel Heyard and Leonhard Held

No e9nw2, MetaArXiv from Center for Open Science

Abstract: Recent large-scale replication projects (RPs) have estimated alarmingly low replicability rates have been estimated. Within these RPs, the original-replication study pairs can vary substantially with respect to aspects of study design, outcome measures, and descriptive features of both the original and replication study population and study team. When broader claims about the replicability of an entire field are based on such heterogeneous data, it becomes imperative to conduct a rigorous analysis of the heterogeneity among study pairs included in the RP. Methodology from the meta-analysis literature provides an approach for quantifying the heterogeneity present in RPs, as additive or multiplicative parameter. Meta-analysis methodology further allows for an investigation of the sources of the heterogeneity through meta-regressions. Notably, we propose the use of location-scale meta-regressions as a means to directly relate the identified characteristics with shrinkage (represented by the location) and the heterogeneity variance (represented by the scale). The proposed methodology is illustrated using data from the Reproducibility Project Psychology and the Reproducibility Project Experimental Economics. All analysis scripts and data are available online.

Date: 2024-01-29
New Economics Papers: this item is included in nep-ppm
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
https://osf.io/download/65b7f6261a30ef0024c343b0/

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:osf:metaar:e9nw2

DOI: 10.31219/osf.io/e9nw2

Access Statistics for this paper

More papers in MetaArXiv from Center for Open Science
Bibliographic data for series maintained by OSF ().