The Horseshoe-Like Regularization for Feature Subset Selection

Anindya Bhadra (), Jyotishka Datta (), Nicholas G. Polson () and Brandon T. Willard ()
Additional contact information
Anindya Bhadra: Purdue University
Jyotishka Datta: University of Arkansas
Nicholas G. Polson: The University of Chicago Booth School of Business
Brandon T. Willard: The University of Chicago Booth School of Business

Sankhya B: The Indian Journal of Statistics, 2021, vol. 83, issue 1, No 9, 185-214

Abstract: Abstract Feature subset selection arises in many high-dimensional applications of statistics, such as compressed sensing and genomics. The ℓ0 penalty is ideal for this task, the caveat being it requires the NP-hard combinatorial evaluation of all models. A recent area of considerable interest is to develop efficient algorithms to fit models with a non-convex ℓγ penalty for γ ∈ (0,1), which results in sparser models than the convex ℓ1 or lasso penalty, but is harder to fit. We propose an alternative, termed the horseshoe regularization penalty for feature subset selection, and demonstrate its theoretical and computational advantages. The distinguishing feature from existing non-convex optimization approaches is a full probabilistic representation of the penalty as the negative of the logarithm of a suitable prior, which in turn enables efficient expectation-maximization and local linear approximation algorithms for optimization and MCMC for uncertainty quantification. In synthetic and real data, the resulting algorithms provide better statistical performance, and the computation requires a fraction of time of state-of-the-art non-convex solvers.

Keywords: Bayes regularization; feature selection; horseshoe estimator; non-convex regularization; scale mixtures.; Primary 62F15; Secondary 62J07; 62C10 (search for similar items in EconPapers)
Date: 2021
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
http://link.springer.com/10.1007/s13571-019-00217-7 Abstract (text/html)
Access to the full text of the articles in this series is restricted.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:spr:sankhb:v:83:y:2021:i:1:d:10.1007_s13571-019-00217-7

Ordering information: This journal article can be ordered from
http://www.springer.com/statistics/journal/13571

DOI: 10.1007/s13571-019-00217-7

Access Statistics for this article

Sankhya B: The Indian Journal of Statistics is currently edited by Dipak Dey

More articles in Sankhya B: The Indian Journal of Statistics from Springer, Indian Statistical Institute
Bibliographic data for series maintained by Sonal Shukla () and Springer Nature Abstracting and Indexing ().