The Anatomy of Out-of-Sample Forecasting Accuracy

Daniel Borup, Philippe Goulet Coulombe, Erik Christian Schütte, David E. Rapach and Sander Schwenk-Nebbe ()

No 2022-16b, FRB Atlanta Working Paper from Federal Reserve Bank of Atlanta

Abstract: We introduce the performance-based Shapley value (PBSV) to measure the contributions of individual predictors to the out-of-sample loss for time-series forecasting models. Our new metric allows a researcher to anatomize out-of-sample forecasting accuracy, thereby providing valuable information for interpreting time-series forecasting models. The PBSV is model agnostic—so it can be applied to any forecasting model, including "black box" models in machine learning, and it can be used for any loss function. We also develop the TS-Shapley-VI, a version of the conventional Shapley value that gauges the importance of predictors for explaining the in-sample predictions in the entire sequence of fitted models that generates the time series of out-of-sample forecasts. We then propose the model accordance score to compare predictor ranks based on the TS-Shapley-VI and PBSV, thereby linking the predictors' in-sample importance to their contributions to out-of-sample forecasting accuracy. We illustrate our metrics in an application forecasting US inflation.

Keywords: model interpretation; Shapley value; predictor importance; loss function; machine learning; inflation (search for similar items in EconPapers)
JEL-codes: C22 C45 C52 C53 E31 E37 (search for similar items in EconPapers)
Pages: 36
Date: 2024-02-21
New Economics Papers: this item is included in nep-big, nep-cmp, nep-ets, nep-for and nep-gth
References: View references in EconPapers View complete reference list from CitEc
Citations:

Published in 2024

Downloads: (external link)
https://www.atlantafed.org/-/media/documents/resea ... casting-accuracy.pdf (application/pdf)

Related works:
Working Paper: The Anatomy of Out-of-Sample Forecasting Accuracy (2022)
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:fip:fedawp:97785

Ordering information: This working paper can be ordered from

DOI: 10.29338/wp2022-16b

Access Statistics for this paper

More papers in FRB Atlanta Working Paper from Federal Reserve Bank of Atlanta Contact information at EDIRC.
Bibliographic data for series maintained by Rob Sarwark ().