Choosing a Strictly Proper Scoring Rule

Edgar C. Merkle () and Mark Steyvers ()
Additional contact information
Edgar C. Merkle: Department of Psychological Sciences, University of Missouri, Columbia, Missouri 65211
Mark Steyvers: Department of Cognitive Sciences, University of California, Irvine, Irvine, California 92697

Decision Analysis, 2013, vol. 10, issue 4, 292-304

Abstract: Strictly proper scoring rules, including the Brier score and the logarithmic score, are standard metrics by which probability forecasters are assessed and compared. Researchers often find that one's choice of strictly proper scoring rule has minimal impact on one's conclusions, but this conclusion is typically drawn from a small set of popular rules. In the context of forecasting world events, we use a recently proposed family of proper scoring rules to study the properties of a wide variety of strictly proper rules. The results indicate that conclusions vary greatly across different scoring rules, so that one's choice of scoring rule should be informed by the forecasting domain. We then describe strategies for choosing a scoring rule that meets the needs of the forecast consumer, considering three unique families of proper scoring rules.

Keywords: proper scoring rule; forecasting; subjective probability; Brier score (search for similar items in EconPapers)
Date: 2013
References: View references in EconPapers View complete reference list from CitEc
Citations: View citations in EconPapers (7)

Downloads: (external link)
http://dx.doi.org/10.1287/deca.2013.0280 (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:inm:ordeca:v:10:y:2013:i:4:p:292-304

Access Statistics for this article

More articles in Decision Analysis from INFORMS Contact information at EDIRC.
Bibliographic data for series maintained by Chris Asher ().