Out-Of-Sample Comparisons of Overfit Models

Gray Calhoun

Staff General Research Papers Archive from Iowa State University, Department of Economics

Abstract: This paper uses dimension asymptotics to study why overfit linear regression models should be compared out-of-sample; we let the number of predictors used by the larger model increase with the number of observations so that their ratio remains uniformly positive. Our analysis gives a theoretical motivation for using out-of-sample (OOS) comparisons: the DMW OOS test allows a forecaster to conduct inference about the expected future accuracy of his or her models when one or both is overfit. We show analytically and through Monte Carlo that standard full-sample test statistics can not test hypotheses about this performance. Our paper also shows that popular test and training sample sizes may give misleading results if researchers are concerned about overfit. We show that P 2 /T must converge to zero for theDMW test to give valid inference about the expected forecast accuracy, otherwise the test measures the accuracy of the estimates constructed using only the training sample. In empirical research, P is typically much larger than this. Our simulations indicate that using large values of P with the DMW test gives undersized tests with low power, so this practice may favor simple benchmark models too much.

Keywords: Generalization Error; Forecasting; ModelSelection; t-test; Dimension Asymptotics (search for similar items in EconPapers)
JEL-codes: C01 C12 C22 C52 C53 (search for similar items in EconPapers)
Date: 2014-03-28
New Economics Papers: this item is included in nep-cba, nep-cis, nep-ecm, nep-ets and nep-for
References: View references in EconPapers View complete reference list from CitEc
Citations: View citations in EconPapers (13)

Downloads: (external link)
http://www2.econ.iastate.edu/papers/p12462-2014-03-28.pdf (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:isu:genres:32462

Access Statistics for this paper

More papers in Staff General Research Papers Archive from Iowa State University, Department of Economics Iowa State University, Dept. of Economics, 260 Heady Hall, Ames, IA 50011-1070. Contact information at EDIRC.
Bibliographic data for series maintained by Curtis Balmer (econwebmaster@iastate.edu).