Subset Selection with Shrinkage: Sparse Linear Modeling When the SNR Is Low

Rahul Mazumder (), Peter Radchenko () and Antoine Dedieu ()
Additional contact information
Rahul Mazumder: Massachusetts Institute of Technology, Cambridge, Massachusetts 02139
Peter Radchenko: University of Sydney, Camperdown, New South Wales 2006, Australia
Antoine Dedieu: Vicarious AI, Union City, California 94587

Operations Research, 2023, vol. 71, issue 1, 129-147

Abstract: We study a seemingly unexpected and relatively less understood overfitting aspect of a fundamental tool in sparse linear modeling—best subset selection—which minimizes the residual sum of squares subject to a constraint on the number of nonzero coefficients. Whereas the best subset selection procedure is often perceived as the “gold standard” in sparse learning when the signal-to-noise ratio (SNR) is high, its predictive performance deteriorates when the SNR is low. In particular, it is outperformed by continuous shrinkage methods, such as ridge regression and the Lasso. We investigate the behavior of best subset selection in the high-noise regimes and propose an alternative approach based on a regularized version of the least-squares criterion. Our proposed estimators (a) mitigate, to a large extent, the poor predictive performance of best subset selection in the high-noise regimes; and (b) perform favorably, while generally delivering substantially sparser models, relative to the best predictive models available via ridge regression and the Lasso. We conduct an extensive theoretical analysis of the predictive properties of the proposed approach and provide justification for its superior predictive performance relative to best subset selection when the noise level is high. Our estimators can be expressed as solutions to mixed-integer second-order conic optimization problems and, hence, are amenable to modern computational tools from mathematical optimization.

Keywords: Machine Learning and Data Science; high-dimensional statistics; sparse regression; best subset selection; integer programming; error bounds; oracle inequalities; signal-to-noise ratio (search for similar items in EconPapers)
Date: 2023
References: Add references at CitEc
Citations:

Downloads: (external link)
http://dx.doi.org/10.1287/opre.2022.2276 (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:inm:oropre:v:71:y:2023:i:1:p:129-147

Access Statistics for this article

More articles in Operations Research from INFORMS Contact information at EDIRC.
Bibliographic data for series maintained by Chris Asher ().