The Science of Using Science: Towards an Understanding of the Threats to Scaling Experiments

Omar Al-Ubaydli, John List and Dana Suskind

No 25848, NBER Working Papers from National Bureau of Economic Research, Inc

Abstract: Policymakers are increasingly turning to insights gained from the experimental method as a means of informing public policies. Whether—and to what extent—insights from a research study scale to the level of the broader public is, in many situations, based on blind faith. This scale-up problem can lead to a vast waste of resources, a missed opportunity to improve people’s lives, and a diminution in the public’s trust in the scientific method’s ability to contribute to policymaking. This study provides a theoretical lens to deepen our understanding of the science of how to use science. Through a simple model, we highlight three elements of the scale-up problem: (1) when does evidence become actionable (appropriate statistical inference); (2) properties of the population; and (3) properties of the situation. We argue that until these three areas are fully understood and recognized by researchers and policymakers, the threats to scalability will render any scaling exercise as particularly vulnerable. In this way, our work represents a challenge to empiricists to estimate the nature and extent of how important the various threats to scalability are in practice, and to implement those in their original research.

JEL-codes: C9 C90 C91 C92 C93 D03 (search for similar items in EconPapers)
Date: 2019-05
New Economics Papers: this item is included in nep-exp and nep-hpe
Note: DEV ED PE TWP
References: View references in EconPapers View complete reference list from CitEc
Citations: View citations in EconPapers (26)

Downloads: (external link)
http://www.nber.org/papers/w25848.pdf (application/pdf)

Related works:
Working Paper: The science of using science: Towards an understanding of the threats to scaling experiments (2019)
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:nbr:nberwo:25848

Ordering information: This working paper can be ordered from
http://www.nber.org/papers/w25848

Access Statistics for this paper

More papers in NBER Working Papers from National Bureau of Economic Research, Inc National Bureau of Economic Research, 1050 Massachusetts Avenue Cambridge, MA 02138, U.S.A.. Contact information at EDIRC.
Bibliographic data for series maintained by ().