Optimal Out-of-Sample Forecast Evaluation under Stationarity

Filip Stanek

CERGE-EI Working Papers from The Center for Economic Research and Graduate Education - Economics Institute, Prague

Abstract: It is common practice to split time-series into in-sample and pseudo out-of-sample segments and to estimate the out-of-sample loss of a given statistical model by evaluating forecasting performance over the pseudo out-of-sample segment. We propose an alternative estimator of the out-of-sample loss which, contrary to conventional wisdom, utilizes both measured in-sample and out-of-sample performance via a carefully constructed system of affine weights. We prove that, provided that the time-series is stationary, the proposed estimator is the best linear unbiased estimator of the out-of-sample loss and outperforms the conventional estimator in terms of sampling variance. Applying the optimal estimator to Diebold-Mariano type tests of predictive ability leads to a substantial power gain without worsening finite sample level distortions. An extensive evaluation on real world time-series from the M4 forecasting competition confirms the superiority of the proposed estimator and also demonstrates a substantial robustness to the violation of the underlying assumption of stationarity.

Keywords: loss estimation; forecast evaluation; cross-validation; model selection (search for similar items in EconPapers)
JEL-codes: C22 C52 C53 (search for similar items in EconPapers)
Date: 2021-11
New Economics Papers: this item is included in nep-ecm, nep-ets, nep-for and nep-ore
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
http://www.cerge-ei.cz/pdf/wp/Wp712.pdf (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:cer:papers:wp712

Access Statistics for this paper

More papers in CERGE-EI Working Papers from The Center for Economic Research and Graduate Education - Economics Institute, Prague Contact information at EDIRC.
Bibliographic data for series maintained by Lucie Vasiljevova ().