Optimal out‐of‐sample forecast evaluation under stationarity

Filip Staněk

Journal of Forecasting, 2023, vol. 42, issue 8, 2249-2279

Abstract: It is a common practice to split a time series into an in‐sample and pseudo‐out‐of‐sample segments and estimate the out‐of‐sample loss for a given statistical model by evaluating forecasting performance over the pseudo‐out‐of‐sample segment. We propose an alternative estimator of the out‐of‐sample loss, which, contrary to the conventional wisdom, utilizes criteria measured both in‐ and out‐of‐sample via a carefully constructed system of affine weights. We prove that, provided that the time series is stationary, the proposed estimator is the best linear unbiased estimator of the out‐of‐sample loss and outperforms the conventional estimator in terms of sampling variability. Application of the optimal estimator to Diebold–Mariano type tests of predictive ability leads to a substantial power gain without increasing finite sample size distortions. An extensive evaluation on real‐world time series from the M4 forecasting competition confirms superiority of the proposed estimator and also demonstrates substantial robustness to violations of the underlying assumption of stationarity.

Date: 2023
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
https://doi.org/10.1002/for.3013

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:wly:jforec:v:42:y:2023:i:8:p:2249-2279

Access Statistics for this article

Journal of Forecasting is currently edited by Derek W. Bunn

More articles in Journal of Forecasting from John Wiley & Sons, Ltd.
Bibliographic data for series maintained by Wiley Content Delivery ().