Stability of Experimental Results: Forecasts and Evidence

Stefano DellaVigna and Devin Pope

No 25858, NBER Working Papers from National Bureau of Economic Research, Inc

Abstract: How robust are experimental results to changes in design? And can researchers anticipate which changes matter most? We consider a specific context, a real-effort task with multiple behavioral treatments, and examine the stability along six dimensions: (i) pure replication; (ii) demographics; (iii) geography and culture; (iv) the task; (v) the output measure; (vi) the presence of a consent form. We use rank-order correlation across the treatments as measure of stability, and compare the observed correlation to the one under a benchmark of full stability (which allows for noise), and to expert forecasts. The academic experts expect that the pure replication will be close to perfect, that the results will differ sizably across demographic groups (age/gender/education), and that changes to the task and output will make a further impact. We find near perfect replication of the experimental results, and full stability of the results across demographics, significantly higher than the experts expected. The results are quite different across task and output change, mostly because the task change adds noise to the findings. The results are also stable to the lack of consent. Overall, the full stability benchmark is an excellent predictor of the observed stability, while expert forecasts are not that informative. This suggests that researchers' predictions about external validity may not be as informative as they expect. We discuss the implications of both the methods and the results for conceptual replication.

JEL-codes: C9 C91 C93 (search for similar items in EconPapers)
Date: 2019-05
New Economics Papers: this item is included in nep-exp and nep-ure
Note: LS
References: View references in EconPapers View complete reference list from CitEc
Citations: View citations in EconPapers (1)

Published as Stefano DellaVigna & Devin Pope, 2022. "Stability of Experimental Results: Forecasts and Evidence," American Economic Journal: Microeconomics, vol 14(3), pages 889-925.

Downloads: (external link)
http://www.nber.org/papers/w25858.pdf (application/pdf)

Related works:
Journal Article: Stability of Experimental Results: Forecasts and Evidence (2022)
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:nbr:nberwo:25858

Ordering information: This working paper can be ordered from
http://www.nber.org/papers/w25858

Access Statistics for this paper

More papers in NBER Working Papers from National Bureau of Economic Research, Inc National Bureau of Economic Research, 1050 Massachusetts Avenue Cambridge, MA 02138, U.S.A.. Contact information at EDIRC.
Bibliographic data for series maintained by (wpc@nber.org).