The Sources of Researcher Variation in Economics

Nick Huntington-Klein, Claus Pörtner, Yubraj Acharya, Matus Adamkovic, Joop Adema, Lameck Ondieki Agasa, Imtiaz Ahmad, Mevlude Akbulut-Yuksel, Martin Eckhoff Andresen, David Angenendt, José-Ignacio Antón, Andreu Arenas, Erkmen Giray Aslim, Stanislav Avdeev, Andrew Bacher-Hicks, Bradley Baker, Imesh Nuwan Bandara, Avijit Bansal, David Bartram, Katarzyna Bech-Wysocka (), Christopher Bennett, Andu Berha, Inés Berniell, Moiz Bhai, Shreya Bhattacharya, Markus Bjoerkheim, Jeffrey Bloem, Margaret Brehm, Martín Brun, Florent Buisson, Pralhad Burli, Andrew Camp, Nicola Cerutti, Weiwei Chen, Jeffrey Clement, Matthew Collins, Lee Crawfurd, John Cullinan, Lachlan Deer, Reid Dorsey-Palmateer, Nicolas Duquette, Diego Marino Fages, Grace Falken, Christine Farquharson, Jan Feld, Yevgeniy Feyman, Nathan Fiala, Anne Fitzpatrick, Andrey Fradkin, Evaewero French, Wei Fu, Luca Fumarco, Sebastian Gallegos, Aaron Gamino, Romain Gauriot, Victor Gay (), Savas Gayaker, Jules Gazeaud, Alexandra de Gendre, Gregory Gilpin, Daniele Girardi, Dan Goldhaber, Mark Harris, Blake Heller, Daniel Henderson, Arne Henningsen, Junita Henry, Clément Herman, Øystein Hernæs, Andrew Hill, Felix Holzmeister, Martijn Huysmans, M. Saad Imtiaz, Anil Jain, Niklas Jakobsson, José Kaire, Kalyan Kumar Kameshwara, Daniel Karney, Sie Won Kim, Valentin Klotzbücher, Christoph Kronenberg, Dan Lafave, David Lang, Ryan Lee, Maxime Liégey, Dede Long (), Jan Marcus, Gabriele Mari, Ian Mccarthy, Laura Meinzen-Dick, Erik Merkus, Klaus Miller, Lukas Mogge, S. Murad, Rafiuddin Najam, Elias Naumann, Job Nmadu, Gorkem Turgut Ozer, Jayash Paudel, Filippos Petroulakis, Christian Peukert, Simon Porcher (), Manab Prakash, Andrew Adrian Pua, Todd Pugatch, Daniel Putman, Veeshan Rayamajhee, Obeid Ur Rehman, Maira Reimão, Anna Reuter, Michael Ricks, Fernando Rios-Avila, Abel Rodriguez, Julian Roeckert, Ivan Ropovik, Jayjit Roy, Nicolas Salamanca, Margaret Samahita, Aparna Samudra, Vassiki Sanogo, Orkhan Sariyev, Henning Schaak, Joel Segel, Hans Henrik Sievertsen, Mike Smet, Brock Smith, Lucy Sorensen, Lisa Spantig, Krzysztof Szczygielski, Anirudh Tagat, Huseyin Tastan, Martin Trombetta, Madhavi Venkatesan, Antoine Vernet, Eden Volkov, Gary Wagner, Yue Wang, Zachary Ward, Tom Waters, Ellerie Weber, Stephen Weinberg, Kristina Weißmüller, Christian Westheide, Kevin Williams, Xiaoyang Ye, Jisang Yu, Muhammad Umer Zahid and Raffaele Zanoli
Additional contact information
Klaus Miller: HEC Paris - Ecole des Hautes Etudes Commerciales

Working Papers from HAL

Abstract: We use a rigorous three-stage many-analysts design to assess how different researcher decisions—specifically data cleaning, research design, and the interpretation of a policy question—affect the variation in estimated treatment effects. A total of 146 research teams each completed the same causal inference task three times each: first with few constraints, then using a shared research design, and finally with pre-cleaned data in addition to a specified design. We find that even when analyzing the same data, teams reach different conclusions. In the first stage, the interquartile range (IQR) of the reported policy effect was 3.1 percentage points, with substantial outliers. Surprisingly, the second stage, which restricted research design choices, exhibited slightly higher IQR (4.0 percentage points), largely attributable to imperfect adherence to the prescribed protocol. By contrast, the final stage, featuring standardized data cleaning, narrowed variation in estimated effects, achieving an IQR of 2.4 percentage points. Reported sample sizes also displayed significant convergence under more restrictive conditions, with the IQR dropping from 295,187 in the first stage to 29,144 in the second, and effectively zero by the third. Our findings underscore the critical importance of data cleaning in shaping applied microeconomic results and highlight avenues for future replication efforts.

Keywords: Metascience; Applied econometrics; Causal inference; Research methods (search for similar items in EconPapers)
Date: 2025-02-25
References: Add references at CitEc
Citations:

There are no downloads for this item, see the EconPapers FAQ for hints about obtaining it.

Related works:
Working Paper: The Sources of Researcher Variation in Economics (2025)
Working Paper: The Sources of Researcher Variation in Economics (2025)
Working Paper: The Sources of Researcher Variation in Economics (2025)
Working Paper: The Sources of Researcher Variation in Economics (2025)
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:hal:wpaper:hal-05187084

DOI: 10.2139/ssrn.5152665

Access Statistics for this paper

More papers in Working Papers from HAL
Bibliographic data for series maintained by CCSD ().