Testing for Changes in Forecasting Performance
Pierre Perron () and
No 2018-03, Discussion Papers from Graduate School of Economics, Hitotsubashi University
We consider the issue of forecast failure (or breakdown) and propose methods to assess retrospectively whether a given forecasting model provides forecasts which show evidence of changes with respect to some loss function. We adapt the classical structural change tests to the forecast failure context. First, we recommend that all tests should be carried with a fixed scheme to have best power. This ensures a maximum difference between the fitted in and out-of-sample means of the losses and avoids contamination issues under the rolling and recursive schemes. With a fixed scheme, Giacomini and Rossi's (2009) (GR) test is simply a Wald test for a one-time change in the mean of the total (the in-sample plus out-of-sample) losses at a known break date, say m, the value that separates the in and out-of-sample periods. To alleviate this problem, we consider a variety of tests: maximizing the GR test over all possible values of m within a pre-specified range; a Double sup-Wald (DSW) test which for each m performs a sup-Wald test for a change in the mean of the out-of-sample losses and takes the maximum of such tests over some range; we also propose to work directly with the total loss series to define the Total Loss sup-Wald (TLSW) test and the Total Loss UDmax (TLUD) test. Using extensive simulations, we show that with forecasting models potentially involving lagged dependent variables, the only tests having a monotonic power function for all data-generating processes are the DSW and TLUD tests, constructed with a fixed forecasting window scheme. Some explanations are provided and two empirical applications illustrate the relevance of our findings in practice.
Keywords: forecast failure; non-monotonic power; structural change; out-of-sample method (search for similar items in EconPapers)
JEL-codes: C14 C22 (search for similar items in EconPapers)
New Economics Papers: this item is included in nep-ecm, nep-for, nep-ore and nep-rmg
References: View references in EconPapers View complete reference list from CitEc
Citations: Track citations by RSS feed
Downloads: (external link)
Working Paper: Testing for Changes in Forecasting Performance (2018)
This item may be available elsewhere in EconPapers: Search for items with the same title.
Export reference: BibTeX
RIS (EndNote, ProCite, RefMan)
Persistent link: https://EconPapers.repec.org/RePEc:hit:econdp:2018-03
Access Statistics for this paper
More papers in Discussion Papers from Graduate School of Economics, Hitotsubashi University Contact information at EDIRC.
Bibliographic data for series maintained by Digital Resources Section, Hitotsubashi University Library ().