The Sources of Researcher Variation in Economics

Nick Huntington-Klein, Claus C. Pörtner, Yubraj Acharya, Matus Adamkovic, Joop Adema, Lameck Ondieki Agasa, Imtiaz Ahmad, Mevlude Akbulut-Yuksel, Martin Eckhoff Andresen, David Angenendt, José-Ignacio Antón, Andreu Arenas, Erkmen Giray Aslim, Stanislav Avdeev, Andrew Bacher-Hicks, Bradley Baker, Imesh Nuwan Bandara, Avijit Bansal, David Bartram, Katarzyna Bech-Wysocka, Christopher T. Bennett, Andu Berha, Inés Berniell, Moiz Bhai, Shreya Bhattacharya, Markus Bjoerkheim, Jeffrey R. Bloem, Margaret Brehm, Martín Brun, Florent Buisson, Pralhad H. Burli, Andrew M. Camp, Nicola Cerutti, Weiwei Chen, Jeffrey Clement, Matthew Collins, Lee Crawfurd, John Cullinan, Lachlan Deer, Reid Dorsey-Palmateer, Nicolas Duquette, Diego Marino Fages, Grace Falken, Christine Farquharson, Jan Feld, Yevgeniy Feyman, Nathan Fiala, Anne Fitzpatrick, Andrey Fradkin, Evaewero French, Wei Fu, Luca Fumarco, Sebastian Gallegos, Julio Galárraga, Aaron M. Gamino, Romain Gauriot, Victor Gay, Savas Gayaker, Jules Gazeaud, Alexandra de Gendre, Gregory Gilpin, Daniele Girardi, Dan Goldhaber, Mark N. Harris, Blake H. Heller, Daniel J. Henderson, Arne Henningsen, Junita Henry, Clément Herman, Øystein Hernæs, Andrew Hill, Felix Holzmeister, Martijn Huysmans, M. Saad Imtiaz, Anil Jain, Niklas Jakobsson, José Kaire, Kalyan Kumar Kameshwara, Daniel Karney, Sie Won Kim, Valentin Klotzbücher, Christoph Kronenberg, Dan LaFave, David Lang, Ryan Lee, Maxime Liégey, Dede Long, Jan Marcus, Gabriele Mari, Ian M. McCarthy, Laura Meinzen-Dick, Erik Merkus, Klaus M. Miller, Lukas Mogge, S. M. Woahid Murad, Rafiuddin Najam, Elias Naumann, Job Nmadu, Gorkem Turgut Ozer, Jayash Paudel, Filippos Petroulakis, Christian Peukert, Visa Pitkänen, Simon Porcher, Manab Prakash, Andrew Adrian Pua, Todd Pugatch, Daniel Putman, Veeshan Rayamajhee, Obeid Ur Rehman, Maira Reimao, Anna Reuter, Michael Ricks, Fernando Rios‐Avila, Abel Rodriguez, Julian Roeckert, Ivan Ropovik, Jayjit Roy, Nicolas Salamanca, Margaret Samahita, Aparna Samudra, Vassiki Sanogo, Orkhan Sariyev, Henning Schaak, Joel E. Segel, Hans Henrik Sievertsen, Mike Smet, Brock Smith, Lucy Sorensen, Lisa Spantig, Krzysztof Szczygielski, Anirudh Tagat, Hüseyin Taştan, Martin Trombetta, Madhavi Venkatesan, Antoine Vernet, Eden Volkov, Gary A. Wagner, Yue Wang, Zachary Ward, Tom Waters, Ellerie Weber, Stephen E Weinberg, Kristina S. Weißmüller, Christian Westheide, Kevin Williams, Xiaoyang Ye, Jisang Yu, Muhammad Umer Zahid and Raffaele Zanoli
Additional contact information
Nick Huntington-Klein: Seattle University - Economics & Finance
Claus C. Pörtner: Seattle University - Albers School of Business and Economics; Center for Studies in Demography and Ecology
Yubraj Acharya: Pennsylvania State University - Department of Health Policy and Administration
Matus Adamkovic: University of Jyväskylä
Joop Adema: CESifo (Center for Economic Studies and Ifo Institute), Ifo Institute; University of Innsbruck
Lameck Ondieki Agasa: Kisii University
Imtiaz Ahmad: National University of Sciences and Technology (NUST); World Bank
Mevlude Akbulut-Yuksel: Dalhousie University ; IZA Institute of Labor Economics
Martin Eckhoff Andresen: University of Oslo - Department of Economics
David Angenendt: Technische Universität München (TUM) - TUM School of Management; University of Cambridge - Centre for Business Research (CBR)
José-Ignacio Antón: University of Salamanca
Andreu Arenas: University of Barcelona
Erkmen Giray Aslim: University of Vermont - Department of Economics
Stanislav Avdeev: University of Amsterdam
Andrew Bacher-Hicks: Boston University
Bradley Baker: Temple University - School of Tourism and Hospitality Management
Imesh Nuwan Bandara: Independent
Avijit Bansal: Indian Institute of Management Calcutta
David Bartram: University of Leicester
Katarzyna Bech-Wysocka: Warsaw School of Economics (SGH) - Institute of Econometrics; Group for Research in Applied Economics (GRAPE)
Christopher T. Bennett: RTI International
Andu Berha: University of Alberta
Inés Berniell: Universidad Nacional de La Plata (UNLP)
Shreya Bhattacharya: College of William & Mary - Global Research Institute and Public Policy Progrsm
Markus Bjoerkheim: Mercatus Center - George Mason University
Jeffrey R. Bloem: International Food Policy Research Institute (IFPRI)
Margaret Brehm: Oberlin College - Department of Economics
Martín Brun: Tampere University; Finnish Centre of Excellence in Tax Systems Research (FIT
Florent Buisson: Independent
Pralhad H. Burli: Government of the United States of America - Idaho National Laboratory
Andrew M. Camp: Brown University - Annenberg Institute
Nicola Cerutti: Bank of England
Weiwei Chen: Kennesaw State University
Jeffrey Clement: Carlson School of Management; Augsburg University
Matthew Collins: University of Galway
Lee Crawfurd: Center for Global Development
John Cullinan: University of Galway
Lachlan Deer: Tilburg University - Department of Marketing
Reid Dorsey-Palmateer: Western Washington University
Nicolas Duquette: University of Southern California
Diego Marino Fages: Durham Business School
Grace Falken: University of Washington
Christine Farquharson: Institute for Fiscal Studies (IFS)
Jan Feld: Victoria University of Wellington - Te Herenga Waka - Victoria Business School
Yevgeniy Feyman: Harvard University
Nathan Fiala: University of Connecticut
Anne Fitzpatrick: Ohio State University (OSU)
Andrey Fradkin: Boston University
Evaewero French: Oregon State University - School of Public Policy
Wei Fu: University of Louisville
Luca Fumarco: Masaryk University; IZA Institute of Labor Economics; Massachusetts Institute of Technology (MIT) - Abdul Latif Jameel Poverty Action Lab (J-PAL); Global Labor Organization (GLO)
Sebastian Gallegos: UAI Business School
Julio Galárraga: Universidad de las Américas (UDLA)
Aaron M. Gamino: MTSU
Romain Gauriot: Deakin University
Victor Gay: Toulouse School of Economics; Institute for Advanced Study in Toulouse
Savas Gayaker: Ankara Haci Bayram Veli University
Jules Gazeaud: Université Clermont Auvergne
Alexandra de Gendre: The University of Melbourne, Department of Economics; University of Melbourne - ARC Centre of Excellence for Children and Families over the Life Course; IZA Institute of Labor Economics
Gregory Gilpin: Montana State University - Bozeman
Daniele Girardi: King’s College London
Mark N. Harris: Curtin University
Blake H. Heller: University of Houston
Daniel J. Henderson: University of Alabama - Department of Economics, Finance and Legal Studies; Institute for the Study of Labor (IZA)
Arne Henningsen: University of Copenhagen - Institute of Food and Resource Economics
Junita Henry: Harvard University - Department of Global Health and Social Medicine
Clément Herman: Princeton University
Øystein Hernæs: Ragnar Frisch Centre for Economic Research
Andrew Hill: Montana State University - Bozeman - Department of Agricultural Economics and Economics
Felix Holzmeister: University of Innsbruck - Department of Economics
Martijn Huysmans: Utrecht University - School of Economics; KU Leuven - Faculty of Business and Economics (FBE)
M. Saad Imtiaz: Lahore University of Management Sciences; World Bank
Anil Jain: Board of Governors of the Federal Reserve System
Niklas Jakobsson: Karlstad University
José Kaire: Arizona State University
Kalyan Kumar Kameshwara: University of Nottingham
Daniel Karney: Ohio University
Sie Won Kim: Texas Tech University - Department of Economics
Valentin Klotzbücher: University of Freiburg
Christoph Kronenberg: University of Duisburg-Essen
Dan LaFave: Colby College
David Lang: Stanford University
Ryan Lee: University of La Verne - College of Business & Public Management
Dede Long: Harvey Mudd College
Jan Marcus: Freie Universität, Berlin
Gabriele Mari: Erasmus University Rotterdam
Ian M. McCarthy: Emory University - Department of Economics; National Bureau of Economic Research (NBER)
Laura Meinzen-Dick: Villanova University
Klaus M. Miller: HEC Paris
Lukas Mogge: RWI - Leibniz Institute for Economic Research
S. M. Woahid Murad: Curtin University; Department of Economics, Noakhali Science & Technology University
Rafiuddin Najam: Oregon State University
Elias Naumann: GESIS - Leibniz Institute for the Social Sciences
Job Nmadu: Federal University of Technology, Minna
Gorkem Turgut Ozer: University of New Hampshire
Jayash Paudel: University of Oklahoma - Department of Economics
Filippos Petroulakis: Bank of Greece
Christian Peukert: University of Lausanne - Faculty of Business and Economics (HEC Lausanne)
Visa Pitkänen: Finnish Competition and Consumer Authority
Simon Porcher: Université Paris Dauphine
Manab Prakash: Tribhuvan University
Andrew Adrian Pua: De La Salle University - School of Economics
Todd Pugatch: State University of New York (SUNY) - University at Buffalo; IZA
Daniel Putman: University of Pennsylvania
Veeshan Rayamajhee: New Mexico State University
Obeid Ur Rehman: Toronto Metropolitan University (formerly Ryerson University)
Maira Reimao: Villanova University
Anna Reuter: Heidelberg University
Michael Ricks: University of Nebraska at Lincoln - Department of Economics; National Bureau of Economic Research (NBER)
Fernando Rios‐Avila: Bard College - The Levy Economics Institute
Abel Rodriguez: Tecnológico de Monterrey
Julian Roeckert: RWI - Leibniz Institute for Economic Research
Ivan Ropovik: Charles University; Czech Academy of Sciences
Jayjit Roy: Appalachian State University
Nicolas Salamanca: Melbourne Institute: Applied Economic & Social Research, the University of Melbourne; IZA; University of Melbourne - ARC Centre of Excellence for Children and Families over the Life Course
Margaret Samahita: University College Dublin
Aparna Samudra: RTM Nagpur University - Department of Economics
Vassiki Sanogo: University of Florida
Orkhan Sariyev: University of Hohenheim
Henning Schaak: Universität für Bodenkultur Wien (BOKU)
Joel E. Segel: Pennsylvania State University
Hans Henrik Sievertsen: University of Bristol; VIVE - The Danish Center for Social Science Research
Mike Smet: KU Leuven - Department of Work and Organization Studies
Brock Smith: Montana State University - Billings
Lucy Sorensen: University at Albany (SUNY)
Lisa Spantig: RWTH Aachen University; University of Essex
Krzysztof Szczygielski: University of Warsaw
Anirudh Tagat: Department of Economics, Monk Prayogshala; Monk Prayogshala; Monash University - School of Mathematics
Hüseyin Taştan: Yildiz Technical University
Martin Trombetta: CONICET
Madhavi Venkatesan: Northeastern University - Department of Economics
Antoine Vernet: University College London
Eden Volkov: US Department of Health and Human Services
Gary A. Wagner: University of Louisiana at Lafayette
Yue Wang: University of Canterbury, Economics and Finance, Students
Zachary Ward: Baylor University - Department of Economics
Tom Waters: Institute for Fiscal Studies (IFS); University College London
Ellerie Weber: Icahn School of Medicine at Mount Sinai
Stephen E Weinberg: New York State Department of Health
Kristina S. Weißmüller: VU University Amsterdam - Department of Political Science and Public Administration
Christian Westheide: University of Vienna - Department of Finance; University of Edinburgh Business School; Leibniz Institute for Financial Research SAFE
Kevin Williams: Occidental College
Xiaoyang Ye: Brown University
Jisang Yu: Kansas State University; Korea University - Department of Food & Resource Economics
Muhammad Umer Zahid: University of Connecticut
Raffaele Zanoli: Polytechnic University of Marche

No 1551, HEC Research Papers Series from HEC Paris

Abstract: We use a rigorous three-stage many-analysts design to assess how different researcher decisions—specifically data cleaning, research design, and the interpretation of a policy question—affect the variation in estimated treatment effects. A total of 146 research teams each completed the same causal inference task three times each: first with few constraints, then using a shared research design, and finally with pre-cleaned data in addition to a specified design. We find that even when analyzing the same data, teams reach different conclusions. In the first stage, the interquartile range (IQR) of the reported policy effect was 3.1 percentage points, with substantial outliers. Surprisingly, the second stage, which restricted research design choices, exhibited slightly higher IQR (4.0 percentage points), largely attributable to imperfect adherence to the prescribed protocol. By contrast, the final stage, featuring standardized data cleaning, narrowed variation in estimated effects, achieving an IQR of 2.4 percentage points. Reported sample sizes also displayed significant convergence under more restrictive conditions, with the IQR dropping from 295,187 in the first stage to 29,144 in the second, and effectively zero by the third. Our findings underscore the critical importance of data cleaning in shaping applied microeconomic results and highlight avenues for future replication efforts.

Keywords: Metascience; Applied econometrics; Causal inference; Research methods (search for similar items in EconPapers)
JEL-codes: C10 (search for similar items in EconPapers)
Pages: 72 pages
Date: 2025-02-25
References: Add references at CitEc
Citations:

There are no downloads for this item, see the EconPapers FAQ for hints about obtaining it.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:ebg:heccah:1551

DOI: 10.2139/ssrn.5152665

Access Statistics for this paper

More papers in HEC Research Papers Series from HEC Paris HEC Paris, 1 Rue de la Libération, 78350 Jouy-en-Josas, France. Contact information at EDIRC.
Bibliographic data for series maintained by Antoine Haldemann ().