An aggregated metrics framework for multicriteria model validation using rolling origin evaluation

Stanisław Halkiewicz and Mateusz Stachowicz

Journal of Risk Model Validation

Abstract: This paper extends the rolling origin evaluation framework to model validation in multicriteria settings, where performance must be assessed across several scenarios or forecast targets. We propose three complementary metrics: the weighted sum of errors; the weighted aggregate performance metric; and the combined error and standard deviation metric. These metrics allow users to balance expected accuracy, fairness across scenarios and stability over repeated splits. A stress testing case study illustrates their practical value: the same gross domestic product growth series is forecast under baseline, adverse and prosperity scenarios, with supervisory-style weights reflecting regulatory priorities. The results show how each metric encodes a distinct evaluation philosophy and may recommend a different model depending on whether accuracy, balance or robustness is emphasized. We further introduce correlation-adjusted variants that penalize systemic errors across scenarios, ensuring that models vulnerable to structural shifts are not inadvertently selected. Together, these contributions provide a structured, quantitative framework for risk-aware model selection, supporting applications in finance, economics and other domains where scenario-based evaluation is essential.

References: Add references at CitEc
Citations:

Downloads: (external link)
https://www.risk.net/node/7962347 (text/html)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:rsk:journ5:7962347

Access Statistics for this article

More articles in Journal of Risk Model Validation from Journal of Risk Model Validation
Bibliographic data for series maintained by Thomas Paine ().