Confidence Calibration in a Multiyear Geopolitical Forecasting Competition

Don A. Moore (), Samuel A. Swift (), Angela Minster (), Barbara Mellers (), Lyle Ungar (), Philip Tetlock (), Heather H. J. Yang () and Elizabeth R. Tenney ()
Additional contact information
Don A. Moore: University of California, Berkeley, Berkeley, California 94720
Samuel A. Swift: Betterment, LLC, New York, New York 10010
Angela Minster: University of Pennsylvania, Philadelphia, Pennsylvania 19104
Barbara Mellers: University of Pennsylvania, Philadelphia, Pennsylvania 19104
Lyle Ungar: University of Pennsylvania, Philadelphia, Pennsylvania 19104
Philip Tetlock: University of Pennsylvania, Philadelphia, Pennsylvania 19104
Heather H. J. Yang: Massachusetts Institute of Technology, Cambridge, Massachusetts 02139
Elizabeth R. Tenney: University of Utah, Salt Lake City, Utah 84112

Management Science, 2017, vol. 63, issue 11, 3552-3565

Abstract: This research examines the development of confidence and accuracy over time in the context of forecasting. Although overconfidence has been studied in many contexts, little research examines its progression over long periods of time or in consequential policy domains. This study employs a unique data set from a geopolitical forecasting tournament spanning three years in which thousands of forecasters predicted the outcomes of hundreds of events. We sought to apply insights from research to structure the questions, interactions, and elicitations to improve forecasts. Indeed, forecasters’ confidence roughly matched their accuracy. As information came in, accuracy increased. Confidence increased at approximately the same rate as accuracy, and good calibration persisted. Nevertheless, there was evidence of a small amount of overconfidence (3%), especially on the most confident forecasts. Training helped reduce overconfidence, and team collaboration improved forecast accuracy. Together, teams and training reduced overconfidence to 1%. Our results provide reason for tempered optimism regarding confidence calibration and its development over time in consequential field contexts.

Keywords: confidence; overconfidence; forecasting; prediction (search for similar items in EconPapers)
Date: 2017
References: View references in EconPapers View complete reference list from CitEc
Citations: View citations in EconPapers (6)

Downloads: (external link)
https://doi.org/10.1287/mnsc.2016.2525 (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:inm:ormnsc:v:63:y:2017:i:11:p:3552-3565

Access Statistics for this article

More articles in Management Science from INFORMS Contact information at EDIRC.
Bibliographic data for series maintained by Chris Asher ().