Humans in the Loop: The Next Frontier in the Credibility Revolution

Megan T. Stevenson and Joshua B. Fischman

No 296, I4R Discussion Paper Series from The Institute for Replication (I4R)

Abstract: Something is amiss in empirical economics. Despite the advances of the credibility revolution, published estimates tend to be inflated and overconfident. We argue that this stems from a weakness in the dominant econometric framework: treating the researcher like a calculator that mechanically implements the econometric method. We use several examples to show how properties of estimators change dramatically with humans-in-the-loop. Under plausible assumptions on researcher behavior, lowpower estimators such as instrumental variables exhibit high degrees of bias, even with a first-stage F-statistic of 200. Threshold testing on the first-stage F-statistic can reduce bias, contrary to Angrist and Koles'ar (2024). And standard errors understate uncertainty, since they ignore variation due to researchers' subjective choices. Ignoring the role of humans "in the research loop" can lead to highly biased and unreliable findings. Modifying econometric practices to address the human factor is a critical frontier of the credibility revolution.

Date: 2026
References: Add references at CitEc
Citations:

Downloads: (external link)
https://www.econstor.eu/bitstream/10419/341115/1/I4R-DP296.pdf (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:zbw:i4rdps:296

Access Statistics for this paper

More papers in I4R Discussion Paper Series from The Institute for Replication (I4R)
Bibliographic data for series maintained by ZBW - Leibniz Information Centre for Economics ().