A post-screening diagnostic study for ultrahigh dimensional data

Yaowu Zhang, Yeqing Zhou and Liping Zhu

Journal of Econometrics, 2024, vol. 239, issue 2

Abstract: We propose a consistent lack-of-fit test to assess whether replacing the original ultrahigh dimensional covariates with a given number of linear combinations results in a loss of regression information. To attenuate the spurious correlations that may inflate type-I error rates in high dimensions, we suggest to randomly split the observations into two parts. In the first part, we screen out as many irrelevant covariates as possible. This screening step helps to reduce the ultrahigh dimensionality to a moderate scale. In the second part, we perform a lack-of-fit test for conditional independence in the context of sufficient dimension reduction. In case that some important covariates are missed with a non-ignorable probability in the first screening stage, we introduce a multiple splitting procedure. We further propose a new statistic to test for conditional independence, which is shown to be n-consistent under the null and root-n-consistent under the alternative. We develop a consistent bootstrap procedure to approximate the asymptotic null distribution. The performances of our proposal are evaluated through comprehensive simulations and an empirical analysis of GDP data.

Keywords: Conditional independence; Lack-of-fit test; Sufficient dimension reduction; Sure screening property; Ultrahigh dimensionality (search for similar items in EconPapers)
Date: 2024
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
http://www.sciencedirect.com/science/article/pii/S0304407622001877
Full text for ScienceDirect subscribers only

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:eee:econom:v:239:y:2024:i:2:s0304407622001877

DOI: 10.1016/j.jeconom.2022.09.005

Access Statistics for this article

Journal of Econometrics is currently edited by T. Amemiya, A. R. Gallant, J. F. Geweke, C. Hsiao and P. M. Robinson

More articles in Journal of Econometrics from Elsevier
Bibliographic data for series maintained by Catherine Liu ().