Sure independence screening for ultrahigh dimensional feature space

Jianqing Fan and Jinchi Lv

Journal Of The Royal Statistical Society Series B, 2008, vol. 70, issue 5, pages 849-911

Abstract: Variable selection plays an important role in high dimensional statistical modelling which nowadays appears in many areas and is key to various scientific discoveries. For problems of large scale or dimensionality "p", accuracy of estimation and computational cost are two top concerns. Recently, Candes and Tao have proposed the Dantzig selector using "L"₁-regularization and showed that it achieves the ideal risk up to a logarithmic factor  log ("p"). Their innovative procedure and remarkable result are challenged when the dimensionality is ultrahigh as the factor  log ("p") can be large and their uniform uncertainty principle can fail. Motivated by these concerns, we introduce the concept of sure screening and propose a sure screening method that is based on correlation learning, called sure independence screening, to reduce dimensionality from high to a moderate scale that is below the sample size. In a fairly general asymptotic framework, correlation learning is shown to have the sure screening property for even exponentially growing dimensionality. As a methodological extension, iterative sure independence screening is also proposed to enhance its finite sample performance. With dimension reduced accurately from high to below sample size, variable selection can be improved on both speed and accuracy, and can then be accomplished by a well-developed method such as smoothly clipped absolute deviation, the Dantzig selector, lasso or adaptive lasso. The connections between these penalized least squares methods are also elucidated. Copyright (c) 2008 Royal Statistical Society.

Date: 2008

Downloads: (external link)
http://www.blackwell-synergy.com/doi/abs/10.1111/j.1467-9868.2008.00674.x link to full text (text/html)
Access to full text is restricted to subscribers.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: http://EconPapers.repec.org/RePEc:bla:jorssb:v:70:y:2008:i:5:p:849-911

Ordering information: This journal article can be ordered from
http://www.blackwell ... bs.asp?ref=1369-7412

Access Statistics for this article

Journal Of The Royal Statistical Society Series B is edited by C. Robert and A. T. A. Wood

More articles in Journal Of The Royal Statistical Society Series B from Royal Statistical Society
Series data maintained by Christopher F. Baum ().