The Horseshoe-Like Regularization for Feature Subset Selection
Anindya Bhadra (),
Jyotishka Datta (),
Nicholas G. Polson () and
Brandon T. Willard ()
Additional contact information
Anindya Bhadra: Purdue University
Jyotishka Datta: University of Arkansas
Nicholas G. Polson: The University of Chicago Booth School of Business
Brandon T. Willard: The University of Chicago Booth School of Business
Sankhya B: The Indian Journal of Statistics, 2021, vol. 83, issue 1, No 9, 185-214
Abstract Feature subset selection arises in many high-dimensional applications of statistics, such as compressed sensing and genomics. The ℓ0 penalty is ideal for this task, the caveat being it requires the NP-hard combinatorial evaluation of all models. A recent area of considerable interest is to develop efficient algorithms to fit models with a non-convex ℓγ penalty for γ ∈ (0,1), which results in sparser models than the convex ℓ1 or lasso penalty, but is harder to fit. We propose an alternative, termed the horseshoe regularization penalty for feature subset selection, and demonstrate its theoretical and computational advantages. The distinguishing feature from existing non-convex optimization approaches is a full probabilistic representation of the penalty as the negative of the logarithm of a suitable prior, which in turn enables efficient expectation-maximization and local linear approximation algorithms for optimization and MCMC for uncertainty quantification. In synthetic and real data, the resulting algorithms provide better statistical performance, and the computation requires a fraction of time of state-of-the-art non-convex solvers.
Keywords: Bayes regularization; feature selection; horseshoe estimator; non-convex regularization; scale mixtures.; Primary 62F15; Secondary 62J07; 62C10 (search for similar items in EconPapers)
References: View references in EconPapers View complete reference list from CitEc
Citations: Track citations by RSS feed
Downloads: (external link)
http://link.springer.com/10.1007/s13571-019-00217-7 Abstract (text/html)
Access to the full text of the articles in this series is restricted.
This item may be available elsewhere in EconPapers: Search for items with the same title.
Export reference: BibTeX
RIS (EndNote, ProCite, RefMan)
Persistent link: https://EconPapers.repec.org/RePEc:spr:sankhb:v:83:y:2021:i:1:d:10.1007_s13571-019-00217-7
Ordering information: This journal article can be ordered from
Access Statistics for this article
Sankhya B: The Indian Journal of Statistics is currently edited by Dipak Dey
More articles in Sankhya B: The Indian Journal of Statistics from Springer, Indian Statistical Institute
Bibliographic data for series maintained by Sonal Shukla () and Springer Nature Abstracting and Indexing ().