(In)Credible Development RCTs: False Discoveries, Low Power, and a Solution

Qingyuan Chai, Kevin Lang and Liang Zhong

No 35999, NBER Working Papers from National Bureau of Economic Research, Inc

Abstract: We study the statistical credibility of randomized controlled trials (RCTs) in development economics and the optimal allocation of resources to replication. Using 745 principal-hypothesis tests from 190 articles in leading economics journals, we find that actual t-statistics peak well below the theoretical benchmark of 2.8 expected under 80 percent power, though above values associated with p-hacking. This suggests many RCTs are underpowered. Estimating an empirical Bayes mixture model that accounts for selective publication, we find that approximately 30 percent of published rejections are false discoveries, with average directional power at only 40 percent. At conventional significance thresholds, the posterior probability of a false discovery reaches 66 percent. Doubling original sample sizes lowers the average false discovery rate by five percentage points, whereas tightening the significance threshold to 0.005 yields a larger reduction at the cost of total discoveries. Finally, we characterize a funder’s optimal replication policy under a fixed budget and an evidence standard. Under standard regularity conditions, replications concentrate in an intermediate interval of original test statistics, and the optimal replication sample size is single-peaked in the original t-statistic. Illustrative calibrations show that successful replications reduce aggregate false discovery rates to 1.8–7.8 percent, depending on the evidence standard.

JEL-codes: C12 C18 O10 O12 (search for similar items in EconPapers)
Date: 2026-11
Note: DEV LS
References: Add references at CitEc
Citations:

Downloads: (external link)
http://www.nber.org/papers/w35999.pdf (application/pdf)
Access to the full text is generally limited to series subscribers, however if the top level domain of the client browser is in a developing country or transition economy free access is provided. More information about subscriptions and free access is available at http://www.nber.org/wwphelp.html. Free access is also available to older working papers.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:nbr:nberwo:35999

Ordering information: This working paper can be ordered from
http://www.nber.org/papers/w35999
The price is Paper copy available by mail.

Access Statistics for this paper

More papers in NBER Working Papers from National Bureau of Economic Research, Inc National Bureau of Economic Research, 1050 Massachusetts Avenue Cambridge, MA 02138, U.S.A.. Contact information at EDIRC.
Bibliographic data for series maintained by ().