Tailored Scoring Rules for Probabilities

David J. Johnstone (), Victor Richmond R. Jose () and Robert L. Winkler ()
Additional contact information
David J. Johnstone: School of Business, University of Sydney, Sydney, New South Wales 2006, Australia
Victor Richmond R. Jose: McDonough School of Business, Georgetown University, Washington, DC 20057
Robert L. Winkler: Fuqua School of Business, Duke University, Durham, North Carolina 27708

Decision Analysis, 2011, vol. 8, issue 4, 256-268

Abstract: When scoring rules were first widely used, they were developed as a way to measure the accuracy of probability forecasts ex post. Ex ante, proper scoring rules encourage honestly reported and sharper probabilities, both of which increase the forecaster's expected score. Most applications utilize standard off-the-shelf scoring rules. In the spirit of decision analysis, we develop proper scoring rules that are tailored to specific decision-making problems and to the utility functions of particular decision makers. We show how these rules, which are intended for situations where a decision maker consults an expert to assess a probability, not only encourage honest reporting, but also reward sharpness in a way that aligns the interests of the expert and the decision maker. We also illustrate the generation of tailored scoring rules in numerical form, which is useful when analytical expressions for the tailored rules cannot be obtained or are too complex to be helpful in practice. Finally, we show how these numerical scoring rules can be presented to the expert in graphical or tabular form and suggest that this could be desirable even for standard scoring rules.

Keywords: probability forecasts; probability evaluation; scoring rules; decision analysis; incentives; utility (search for similar items in EconPapers)
Date: 2011
References: View references in EconPapers View complete reference list from CitEc
Citations: View citations in EconPapers (12)

Downloads: (external link)
http://dx.doi.org/10.1287/deca.1110.0216 (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:inm:ordeca:v:8:y:2011:i:4:p:256-268

Access Statistics for this article

More articles in Decision Analysis from INFORMS Contact information at EDIRC.
Bibliographic data for series maintained by Chris Asher ().