Finite-sample and asymptotic analysis of generalization ability with an application to penalized regression

Ning Xu, Jian Hong and Timothy Fisher

MPRA Paper from University Library of Munich, Germany

Abstract: In this paper, we study the generalization ability (GA)---the ability of a model to predict outcomes in new samples from the same population---of the extremum estimators. By adapting the classical concentration inequalities, we propose upper bounds for the empirical out-of-sample prediction error for extremum estimators, which is a function of the in-sample error, the severity of heavy tails, the sample size of in-sample data and model complexity. The error bounds not only serve to measure GA, but also to illustrate the trade-off between in-sample and out-of-sample fit, which is connected to the traditional bias-variance trade-off. Moreover, the bounds also reveal that the hyperparameter K, the number of folds in $K$-fold cross-validation, cause the bias-variance trade-off for cross-validation error, which offers a route to hyperparameter optimization in terms of GA. As a direct application of GA analysis, we implement the new upper bounds in penalized regression estimates for both n>p and n

Keywords: generalization ability; upper bound of generalization error; penalized regression; bias-variance trade-off; lasso; high-dimensional data; cross-validation; $\mathcal{L}_2$ difference between penalized and unpenalized regression (search for similar items in EconPapers)
JEL-codes: C13 C52 C55 (search for similar items in EconPapers)
Date: 2016-09-10
New Economics Papers: this item is included in nep-ecm and nep-sog
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
https://mpra.ub.uni-muenchen.de/73622/1/MPRA_paper_73622.pdf original version (application/pdf)
https://mpra.ub.uni-muenchen.de/73657/1/MPRA_paper_73657.pdf revised version (application/pdf)

Related works:
Working Paper: Finite-sample and asymptotic analysis of generalization ability with an application to penalized regression (2016)
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:pra:mprapa:73622

Access Statistics for this paper

More papers in MPRA Paper from University Library of Munich, Germany Ludwigstraße 33, D-80539 Munich, Germany. Contact information at EDIRC.
Bibliographic data for series maintained by Joachim Winter ().