Testing for Changes in Forecasting Performance

Pierre Perron and Yohei Yamamoto

Journal of Business & Economic Statistics, 2021, vol. 39, issue 1, 148-165

Abstract: We consider the issue of forecast failure (or breakdown) and propose methods to assess retrospectively whether a given forecasting model provides forecasts which show evidence of changes with respect to some loss function. We adapt the classical structural change tests to the forecast failure context. First, we recommend that all tests should be carried with a fixed scheme to have best power. This ensures a maximum difference between the fitted in and out-of-sample means of the losses and avoids contamination issues under the rolling and recursive schemes. With a fixed scheme, Giacomini and Rossi’s (GR) test is simply a Wald test for a one-time change in the mean of the total (the in-sample plus out-of-sample) losses at a known break date, say m, the value that separates the in and out-of-sample periods. To alleviate this problem, we consider a variety of tests: maximizing the GR test over values of m within a prespecified range; a Double sup-Wald (DSW) test which for each m performs a sup-Wald test for a change in the mean of the out-of-sample losses and takes the maximum of such tests over some range; we also propose to work directly with the total loss series to define the Total Loss sup-Wald and Total Loss UDmax (TLUD) tests. Using theoretical analyses and simulations, we show that with forecasting models potentially involving lagged dependent variables, the only tests having a monotonic power function for all data-generating processes considered are the DSW and TLUD tests, constructed with a fixed forecasting window scheme. Some explanations are provided and empirical applications illustrate the relevance of our findings in practice. Supplementary materials for this article are available online.

Date: 2021
References: Add references at CitEc
Citations: View citations in EconPapers (8)

Downloads: (external link)
http://hdl.handle.net/10.1080/07350015.2019.1641410 (text/html)
Access to full text is restricted to subscribers.

Related works:
Working Paper: Testing for Changes in Forecasting Performance (2019)
Working Paper: Testing for Changes in Forecasting Performance (2018)
Working Paper: Testing for Changes in Forecasting Performance (2018)
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:taf:jnlbes:v:39:y:2021:i:1:p:148-165

Ordering information: This journal article can be ordered from
http://www.tandfonline.com/pricing/journal/UBES20

DOI: 10.1080/07350015.2019.1641410

Access Statistics for this article

Journal of Business & Economic Statistics is currently edited by Eric Sampson, Rong Chen and Shakeeb Khan

More articles in Journal of Business & Economic Statistics from Taylor & Francis Journals
Bibliographic data for series maintained by Chris Longhurst ().