Scoring rules and performance, new analysis of expert judgment data

Gabriela F. Nane and Roger M. Cooke

Futures & Foresight Science, 2024, vol. 6, issue 4

Abstract: A review of scoring rules highlights the distinction between rewarding honesty and rewarding quality. This motivates the introduction of a scale‐invariant version of the Continuous Ranked Probability Score (CRPS) which enables statistical accuracy (SA) testing based on an exact rather than an asymptotic distribution of the density of convolutions. A recent data set of 6761 expert probabilistic forecasts for questions for which the actual values are known is used to compare performance. New insights include that (a) variance due to assessed variables dominates variance due to experts, (b) performance on mean absolute percentage error (MAPE) is weakly related to SA (c) scale‐invariant CRPS combinations compete with the Classical Model (CM) on SA and MAPE, and (d) CRPS is more forgiving with regard to SA than the CM as CRPS is insensitive to location bias.

Date: 2024
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
https://doi.org/10.1002/ffo2.189

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:wly:fufsci:v:6:y:2024:i:4:n:e189

Access Statistics for this article

More articles in Futures & Foresight Science from John Wiley & Sons
Bibliographic data for series maintained by Wiley Content Delivery ().