Variable Selection

Wolfgang Karl Härdle and Zdeněk Hlávka
Additional contact information
Wolfgang Karl Härdle: Humboldt-Universität zu Berlin, C.A.S.E. Centre f. Appl. Stat. & Econ. School of Business and Economics
Zdeněk Hlávka: Charles University in Prague, Faculty of Mathematics and Physics Department of Statistics

Chapter Chapter 9 in Multivariate Statistics, 2015, pp 157-165 from Springer

Abstract: Abstract We have already remarked that multicollinearity, i.e., nearly linearly Multicollinearity dependent columns in the design matrix, may increase the variance of the estimators $$\hat{\beta }_{i}$$ . For simplicity of presentation, we will assume throughout this section that the response is centered and predictor variables are standardized. More formally, Zvára (2008, Theorem 11.1 ) observes in the linear model ( 8.1 ) that $$\displaystyle{\mathop{\mathrm{\mathsf{E}}}\nolimits \|\hat{\beta }\|^{2} =\|\beta \| ^{2} +\sigma ^{2}\mathop{ \mathrm{\text{tr}}}\nolimits (\mathcal{X}^{\top }\mathcal{X})^{-1}}$$ and $$\displaystyle{\mathop{\mathrm{\mathsf{E}}}\nolimits \|\hat{Y }\|^{2} =\| \mathcal{X}\beta \|^{2} +\sigma ^{2}\mathop{ \mathrm{\text{rank}}}\nolimits (\mathcal{X}).}$$ It follows that multicollinearity does not affect the fitted values $$\hat{Y } = \mathcal{X}\hat{\beta }$$ Fitted values because the expectation of its squared length depends only on σ 2 and the rank of the model matrix $$\mathcal{X}$$ . On the other hand, the expectation of the squared length of the estimator $$\hat{\beta }$$ depends on the term $$\mathop{\mathrm{\text{tr}}}\nolimits (\mathcal{X}^{\top }\mathcal{X})^{-1} =\sum \lambda _{ i}^{-1}$$ , where λ i are the eigenvalues of $$\mathcal{X}^{\top }\mathcal{X}$$ . If the columns of $$\mathcal{X}$$ are nearly dependent, some of these eigenvalues may be very small and $$\mathop{\mathrm{\mathsf{E}}}\nolimits \|\hat{\beta }\|^{2}$$ then may become very large even if, technically, the design matrix still has full rank.

Keywords: Tuning Parameter; Variance Inflation Factor; Design Matrix; Ridge Regression; Estimate Regression Coefficient (search for similar items in EconPapers)
Date: 2015
References: Add references at CitEc
Citations:

There are no downloads for this item, see the EconPapers FAQ for hints about obtaining it.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:spr:sprchp:978-3-642-36005-3_9

Ordering information: This item can be ordered from
http://www.springer.com/9783642360053

DOI: 10.1007/978-3-642-36005-3_9

Access Statistics for this chapter

More chapters in Springer Books from Springer
Bibliographic data for series maintained by Sonal Shukla () and Springer Nature Abstracting and Indexing ().