Evaluating the discrimination ability of proper multi-variate scoring rules

Carol Alexander, M. Coulon, Y. Han and X. Meng
Additional contact information
M. Coulon: University of Sussex Business School
Y. Han: University of Sussex Business School
X. Meng: University of Sussex Business School

Annals of Operations Research, 2024, vol. 334, issue 1, No 31, 857-883

Abstract: Abstract Proper scoring rules are commonly applied to quantify the accuracy of distribution forecasts. Given an observation they assign a scalar score to each distribution forecast, with the lowest expected score attributed to the true distribution. The energy and variogram scores are two rules that have recently gained some popularity in multivariate settings because their computation does not require a forecast to have parametric density function and so they are broadly applicable. Here we conduct a simulation study to compare the discrimination ability between the energy score and three variogram scores. Compared with other studies, our simulation design is more realistic because it is supported by a historical data set containing commodity prices, currencies and interest rates, and our data generating processes include a diverse selection of models with different marginal distributions, dependence structure, and calibration windows. This facilitates a comprehensive comparison of the performance of proper scoring rules in different settings. To compare the scores we use three metrics: the mean relative score, error rate and a generalized discrimination heuristic. Overall, we find that the variogram score with parameter $$p=0.5$$ p = 0.5 outperforms the energy score and the other two variogram scores.

Keywords: Multivariate forecasting; Proper scoring rules; Discrimination heuristic; Energy score; Variogram score (search for similar items in EconPapers)
Date: 2024
References: Add references at CitEc
Citations:

Downloads: (external link)
http://link.springer.com/10.1007/s10479-022-04611-9 Abstract (text/html)
Access to the full text of the articles in this series is restricted.

Related works:
Working Paper: Evaluating the Discrimination Ability of Proper Multivariate Scoring Rules (2021)
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:spr:annopr:v:334:y:2024:i:1:d:10.1007_s10479-022-04611-9

Ordering information: This journal article can be ordered from
http://www.springer.com/journal/10479

DOI: 10.1007/s10479-022-04611-9

Access Statistics for this article

Annals of Operations Research is currently edited by Endre Boros

More articles in Annals of Operations Research from Springer
Bibliographic data for series maintained by Sonal Shukla () and Springer Nature Abstracting and Indexing ().