The Anatomy of Out-of-Sample Forecasting Accuracy

Daniel Borup, Philippe Goulet Coulombe, Erik Christian Schütte, David E. Rapach and Sander Schwenk-Nebbe ()

No 2022-16, FRB Atlanta Working Paper from Federal Reserve Bank of Atlanta

Abstract: We develop metrics based on Shapley values for interpreting time-series forecasting models, including “black-box” models from machine learning. Our metrics are model agnostic, so that they are applicable to any model (linear or nonlinear, parametric or nonparametric). Two of the metrics, iShapley-VI and oShapley-VI, measure the importance of individual predictors in fitted models for explaining the in-sample and out-of-sample predicted target values, respectively. The third metric is the performance-based Shapley value (PBSV), our main methodological contribution. PBSV measures the contributions of individual predictors in fitted models to the out-of-sample loss and thereby anatomizes out-of-sample forecasting accuracy. In an empirical application forecasting US inflation, we find important discrepancies between individual predictor relevance according to the in-sample iShapley-VI and out-of-sample PBSV. We use simulations to analyze potential sources of the discrepancies, including overfitting, structural breaks, and evolving predictor volatilities.

Keywords: variable importance; out-of-sample performance; Shapley value; loss function; machine learning; inflation (search for similar items in EconPapers)
JEL-codes: C22 C45 C53 E37 G17 (search for similar items in EconPapers)
Pages: 54
Date: 2022-11-07
New Economics Papers: this item is included in nep-big, nep-cmp, nep-ecm, nep-ets and nep-for
References: Add references at CitEc
Citations: View citations in EconPapers (4)

Published in 2022

Downloads: (external link)
https://www.atlantafed.org/research/publications/w ... asting-accuracy.aspx (application/pdf)

Related works:
Working Paper: The Anatomy of Out-of-Sample Forecasting Accuracy (2024)
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:fip:fedawp:94993

Ordering information: This working paper can be ordered from

DOI: 10.29338/wp2022-16

Access Statistics for this paper

More papers in FRB Atlanta Working Paper from Federal Reserve Bank of Atlanta Contact information at EDIRC.
Bibliographic data for series maintained by Rob Sarwark ().