The Influence of Hidden Researcher Decisions in Applied Microeconomics

Nick Huntington-Klein, Andreu Arenas, Emily Beam, Marco Bertoni, Jeffrey Bloem, Pralhad Burli (), Naibin Chen, Paul Greico (), Godwin Ekpe, Todd Pugatch, Martin Saavedra () and Yaniv Stopnitzky
Additional contact information
Pralhad Burli: Idaho National Laboratory
Naibin Chen: Pennsylvania State University
Paul Greico: Pennsylvania State University
Yaniv Stopnitzky: University of San Francisco

No 13233, IZA Discussion Papers from Institute of Labor Economics (IZA)

Abstract: Researchers make hundreds of decisions about data collection, preparation, and analysis in their research. We use a many-analysts approach to measure the extent and impact of these decisions. Two published causal empirical results are replicated by seven replicators each. We find large differences in data preparation and analysis decisions, many of which would not likely be reported in a publication. No two replicators reported the same sample size. Statistical significance varied across replications, and for one of the studies the effect's sign varied as well. The standard deviation of estimates across replications was 3-4 times the typical reported standard error.

Keywords: metascience; replication; research (search for similar items in EconPapers)
JEL-codes: B41 C10 C81 (search for similar items in EconPapers)
Pages: 42 pages
Date: 2020-05
New Economics Papers: this item is included in nep-sog and nep-tid
References: Add references at CitEc
Citations: View citations in EconPapers (16)

Published - published in: Economic Inquiry, 2021, 59 (3), 944 - 960 https://doi.org/10.1111/ecin.12992

Downloads: (external link)
https://docs.iza.org/dp13233.pdf (application/pdf)

Related works:
Journal Article: The influence of hidden researcher decisions in applied microeconomics (2021)
Working Paper: The Influence of Hidden Researcher Decisions in Applied Microeconomics (2020)
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:iza:izadps:dp13233

Ordering information: This working paper can be ordered from
IZA, Margard Ody, P.O. Box 7240, D-53072 Bonn, Germany

Access Statistics for this paper

More papers in IZA Discussion Papers from Institute of Labor Economics (IZA) IZA, P.O. Box 7240, D-53072 Bonn, Germany. Contact information at EDIRC.
Bibliographic data for series maintained by Holger Hinte ().