Out-of-sample comparisons of overfit models

Gray Calhoun

ISU General Staff Papers from Iowa State University, Department of Economics

Abstract: This paper uses dimension asymptotics to study why overfit linear regression models should be compared out-of-sample; we let the number of predictors used by the larger model increase with the number of observations so that their ratio remains uniformly positive. Our analysis gives a theoretical motivation for using out-of-sample (OOS) comparisons: the DMW OOS test allows a forecaster to conduct inference about the expected future accuracy of his or her models when one or both is overfit. We show analytically and through Monte Carlo that standard full-sample test statistics can not test hypotheses about this performance. Our paper also shows that popular test and training sample sizes may give misleading results if researchers are concerned about overfit. We show that P 2 /T must converge to zero for theDMW test to give valid inference about the expected forecast accuracy, otherwise the test measures the accuracy of the estimates constructed using only the training sample. In empirical research, P is typically much larger than this. Our simulations indicate that using large values of P with the DMW test gives undersized tests with low power, so this practice may favor simple benchmark models too much.

Date: 2014-03-30
References: Add references at CitEc
Citations:

Downloads: (external link)
https://dr.lib.iastate.edu/server/api/core/bitstre ... a6be75fc1dbc/content
Our link check indicates that this URL is bad, the error code is: 403 Forbidden

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:isu:genstf:201403300700001013

Access Statistics for this paper

More papers in ISU General Staff Papers from Iowa State University, Department of Economics Iowa State University, Dept. of Economics, 260 Heady Hall, Ames, IA 50011-1070. Contact information at EDIRC.
Bibliographic data for series maintained by Curtis Balmer ().