Forecasting Social Science: Evidence from 100 Projects

Stefano DellaVigna and Eva Vivalt

No 34493, NBER Working Papers from National Bureau of Economic Research, Inc

Abstract: Forecasts about research findings affect critical scientific decisions, such as what treatments an R&D lab invests in, or which papers a researcher decides to write. But what do we know about the accuracy of these forecasts? We analyze a unique data set of all 100 projects posted on the Social Science Prediction Platform from 2020 to 2024, which received 53,298 forecasts in total, including 66 projects for which we also have results. We show that forecasters, on average, over-estimate treatment effects; however, the average forecast is quite predictive of the actual treatment effect. We also examine differences in accuracy across forecasters. Academics have a slightly higher accuracy than non-academics, but expertise in a field does not increase accuracy. A panel of motivated repeat forecasters has higher accuracy, but this does not extend more broadly to all repeat forecasters. Confidence in the accuracy of one's forecasts is perversely associated with lower accuracy. We also document substantial cross-study correlation in accuracy among forecasters and identify a group of "superforecasters". Finally, we relate our findings to results in the literature as well as to expert forecasts.

JEL-codes: D01 D91 O12 (search for similar items in EconPapers)
Date: 2025-11
New Economics Papers: this item is included in nep-exp and nep-for
Note: AG AP CH DEV ED EEE EFG EH IO ITI LS ME PE POL PR
References: Add references at CitEc
Citations:

Downloads: (external link)
http://www.nber.org/papers/w34493.pdf (application/pdf)
Access to the full text is generally limited to series subscribers, however if the top level domain of the client browser is in a developing country or transition economy free access is provided. More information about subscriptions and free access is available at http://www.nber.org/wwphelp.html. Free access is also available to older working papers.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:nbr:nberwo:34493

Ordering information: This working paper can be ordered from
http://www.nber.org/papers/w34493
The price is Paper copy available by mail.

Access Statistics for this paper

More papers in NBER Working Papers from National Bureau of Economic Research, Inc National Bureau of Economic Research, 1050 Massachusetts Avenue Cambridge, MA 02138, U.S.A.. Contact information at EDIRC.
Bibliographic data for series maintained by ().