Heterogeneity in effect size estimates: Empirical evidence and practical implications

Felix Holzmeister, Magnus Johannesson, Robert Böhm, Anna Dreber, Jürgen Huber and Michael Kirchler

Working Papers from Faculty of Economics and Statistics, Universität Innsbruck

Abstract: A typical empirical study involves choosing a sample, a research design, and an analysis path. Variation in such choices across studies leads to heterogeneity in results that introduce an additional layer of uncertainty not accounted for in reported standard errors and confidence intervals. We provide a framework for studying heterogeneity in the social sciences and divide heterogeneity into population heterogeneity, design heterogeneity, and analytical heterogeneity. We estimate each type's heterogeneity from multi-lab replication studies, prospective meta-analyses of studies varying experimental designs, and multi-analyst studies. Our results suggest that population heterogeneity tends to be relatively small, whereas design and analytical heterogeneity are large. A conservative interpretation of the estimates suggests that incorporating the uncertainty due to heterogeneity would approximately double sample standard errors and confidence intervals. We illustrate that heterogeneity of this magnitude — unless properly accounted for —has severe implications for statistical inference with strongly increased rates of false scientific claims.

Keywords: Conflict; contest; conflict resolution; group decision-making; group identity; alliance; experiment (search for similar items in EconPapers)
Date: 2023
New Economics Papers: this item is included in nep-ecm and nep-exp
References: Add references at CitEc
Citations:

Downloads: (external link)
https://www2.uibk.ac.at/downloads/c9821000/wpaper/2023-17.pdf (application/pdf)

Related works:
Working Paper: Heterogeneity in Effect Size Estimates: Empirical Evidence and Practical Implications (2024)
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:inn:wpaper:2023-17

Access Statistics for this paper

More papers in Working Papers from Faculty of Economics and Statistics, Universität Innsbruck Contact information at EDIRC.
Bibliographic data for series maintained by Judith Courian ( this e-mail address is bad, please contact ).