Can out-of-sample forecast comparisons help prevent overfitting?

Todd Clark

No RWP 00-05, Research Working Paper from Federal Reserve Bank of Kansas City

Abstract: This paper shows that out-of-sample forecast comparisons can help prevent data mining-induced overfitting. The basic results are drawn from simulations of a simple Monte Carlo design and a real data-based design similar to those in Lovell (1983) and Hoover and Perez (1999). In each simulation, a general-to-specific procedure is used to arrive at a model. If the selected specification includes any of the candidate explanatory variables, forecasts from the model are compared to forecasts from a benchmark model that is nested within the selected model. In particular, the competing forecasts are tested for equal MSE and encompassing. The simulations indicate most of the post-sample tests are roughly correctly sized, as long as just the in-sample portion of the data are used in model selection. Moreover, the tests have relatively good power, although some are consistently more powerful than others. The paper concludes with an application, modeling quarterly U.S. inflation.

Keywords: Forecasting (search for similar items in EconPapers)
Date: 2000
New Economics Papers: this item is included in nep-ecm
References: View references in EconPapers View complete reference list from CitEc
Citations: View citations in EconPapers (5)

Downloads: (external link)
https://www.kansascityfed.org/documents/5423/pdf-RWP00-05.pdf (application/pdf)

Related works:
Journal Article: Can out-of-sample forecast comparisons help prevent overfitting? (2004)
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:fip:fedkrw:rwp00-05

Ordering information: This working paper can be ordered from

Access Statistics for this paper

More papers in Research Working Paper from Federal Reserve Bank of Kansas City Contact information at EDIRC.
Bibliographic data for series maintained by Zach Kastens ().