The Sources of Researcher Variation in Economics

Nick Huntington-Klein, Claus Pörtner, Yubraj Acharya, Matus Adamkovic, Joop Adema, Lameck Ondieki Agasa, Imtiaz Ahmad, Mevlude Akbulut-Yuksel, Martin Eckhoff Andresen, David Angenendt, José Ignacio Antón, Andreu Arenas, Erkmen Giray Aslim, Stanislav Avdeev, Andrew Bacher-Hicks, Bradley J. Baker, Imesh Nuwan Bandara, Avijit Bansal, David Bartram, Katarzyna Bech-Wysocka (), Christopher Troy Bennett, Andu Berha, Inés Berniell, Moiz Bhai, Shreya Bhattacharya, Markus Bjoerkheim, Jeffrey R. Bloem, Margaret Brehm, Martín Brun, Florent Buisson, Pralhad Burli, Andrew M. Camp, Nicola Cerutti, Weiwei Chen, Jeffrey Clement, Matthew Collins, Lee Crawfurd, John Cullinan, Lachlan Deer, Reid Dorsey-Palmateer, Nicolas J. Duquette, Diego Marino Fages, Grace Falken, Christine Farquharson, Jan Feld, Yevgeniy Feyman, Nathan Fiala, Anne Fitzpatrick, Andrey Fradkin, Evaewero French, Wei Fu, Luca Fumarco, Sebastian Gallegos, Julio Galárraga, Aaron Gamino, Romain Gauriot, Victor Gay (), Savas Gayaker, Jules Gazeaud, Alexandra de Gendre, Gregory Gilpin, Daniele Girardi, Dan Goldhaber, Mark N. Harris, Blake H. Heller, Daniel Henderson (), Arne Henningsen, Junita Henry, Clément Herman, Øystein Hernæs, Andrew J. Hill, Felix Holzmeister, Martijn Huysmans, M. Saad Imtiaz, Anil K. Jain, Niklas Jakobsson, José Kaire, Kalyan Kumar Kameshwara, Daniel H. Karney, Sie Won Kim, Valentin Klotzbücher, Christoph Kronenberg, Daniel LaFave, David Lang, Ryan Lee, Maxime Liégey, Dede Long (), Jan Marcus, Gabriele Mari, Ian McCarthy, Laura Meinzen-Dick, Erik Merkus, Klaus M. Miller, Lukas Mogge, S. M. Woahid Murad, Rafiuddin Najam, Elias Naumann, Job Nda Nmadu, Gorkem Turgut Ozer, Jayash Paudel, Filippos Petroulakis, Christian Peukert, Visa Pitkänen, Simon Porcher, Manab Prakash, Andrew Adrian Yu Pua, Todd Pugatch, Daniel Putman, Veeshan Rayamajhee, Obeid Ur Rehman, Maira Reimão, Anna Reuter, Michael David Ricks, Fernando Rios-Avila, Abel Rodriguez, Julian Roeckert, Ivan Ropovik, Jayjit Roy, Nicolas Salamanca, Margaret Samahita, Aparna Samudra, Vassiki Sanogo, Orkhan Sariyev, Henning Schaak, Joel E. Segel, Hans Henrik Sievertsen, Mike Smet, Brock Smith, Lucy C. Sorensen, Lisa Spantig, Krzysztof Szczygielski, Anirudh Tagat, Huseyin Tastan, Martin Trombetta, Madhavi Venkatesan, Antoine Vernet, Eden Volkov, Gary A. Wagner, Yue Wang, Zachary Ward, Tom Waters, Ellerie Weber, Stephen E. Weinberg, Kristina S. Weißmüller, Christian Westheide, Kevin Williams, Xiaoyang Ye, Jisang Yu, Muhammad Umer Zahid and Raffaele Zanoli

No 209, I4R Discussion Paper Series from The Institute for Replication (I4R)

Abstract: We use a rigorous three-stage many-analysts design to assess how different researcher decisions-specifically data cleaning, research design, and the interpretation of a policy question-affect the variation in estimated treatment effects. A total of 146 research teams each completed the same causal inference task three times each: first with few constraints, then using a shared research design, and finally with pre-cleaned data in addition to a specified design. We find that even when analyzing the same data, teams reach different conclusions. In the first stage, the interquartile range (IQR) of the reported policy effect was 3.1 percentage points, with substantial outliers. Surprisingly, the second stage, which restricted research design choices, exhibited slightly higher IQR (4.0 percentage points), largely attributable to imperfect adherence to the prescribed protocol. By contrast, the final stage, featuring standardized data cleaning, narrowed variation in estimated effects, achieving an IQR of 2.4 percentage points. Reported sample sizes also displayed significant convergence under more restrictive conditions, with the IQR dropping from 295,187 in the first stage to 29,144 in the second, and effectively zero by the third. Our findings underscore the critical importance of data cleaning in shaping applied microeconomic results and highlight avenues for future replication efforts.

Date: 2025
References: View complete reference list from CitEc
Citations:

Downloads: (external link)
https://www.econstor.eu/bitstream/10419/312260/1/I4R-DP209.pdf (application/pdf)

Related works:
Working Paper: The Sources of Researcher Variation in Economics (2025)
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:zbw:i4rdps:209

Access Statistics for this paper

More papers in I4R Discussion Paper Series from The Institute for Replication (I4R)
Bibliographic data for series maintained by ZBW - Leibniz Information Centre for Economics ().