Continuous Distributions and Measures of Statistical Accuracy for Structured Expert Judgment

Guus Rongen, Gabriela F. Nane, Oswaldo Morales‐Napoles and Roger M. Cooke

Futures & Foresight Science, 2025, vol. 7, issue 2

Abstract: This study evaluates five scoring rules, or measures of statistical accuracy, for assessing uncertainty estimates from expert judgment studies and model forecasts. These rules — the Continuously Ranked Probability Score (CRPS ${CRPS}$), Kolmogorov‐Smirnov (KS ${KS}$), Cramer‐von Mises (CvM ${CvM}$), Anderson Darling (AD ${AD}$), and chi‐square test — were applied to 6864 expert uncertainty estimates from 49 Classical Model (CM) studies. We compared their sensitivity to various biases and their ability to serve as performance‐based weight for expert estimates. Additionally, the piecewise uniform and Metalog distribution were evaluated for their representation of expert estimates because four of the five rules require interpolating the experts' estimates. Simulating biased estimates reveals varying sensitivity of the considered test statistics to these biases. Expert weights derived using one measure of statistical accuracy were evaluated with other measures to assess their performance. The main conclusions are (1) CRPS ${CRPS}$ overlooks important biases, while chi‐square and AD ${AD}$ behave similarly, as do KS ${KS}$ and CvM ${CvM}$. (2) All measures except CRPS ${CRPS}$ agree that performance weighting is superior to equal weighting with respect to statistical accuracy. (3) Neither distributions can effectively predict the position of a removed quantile estimate. These insights show the behavior of different scoring rules for combining uncertainty estimates from expert or models, and extent the knowledge for best‐practices.

Date: 2025
References: Add references at CitEc
Citations:

Downloads: (external link)
https://doi.org/10.1002/ffo2.70009

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:wly:fufsci:v:7:y:2025:i:2:n:e70009

Access Statistics for this article

More articles in Futures & Foresight Science from John Wiley & Sons
Bibliographic data for series maintained by Wiley Content Delivery ().