Projective independence tests in high dimensions: the curses and the cures

Yaowu Zhang and Liping Zhu

Biometrika, 2024, vol. 111, issue 3, 1013-1027

Abstract: SummaryTesting independence between high-dimensional random vectors is fundamentally different from testing independence between univariate random variables. Taking the projection correlation as an example, it suffers from at least three problems. First, it has a high computational complexity of O{n3(p+q)}, where n, p and q are the sample size and dimensions of the random vectors; this limits its usefulness substantially when n is extremely large. Second, the asymptotic null distribution of the projection correlation test is rarely tractable; therefore, random permutations are often suggested as a means of approximating the asymptotic null distribution, which further increases the complexity of implementing independence tests. Third, the power performance of the projection correlation test deteriorates in high dimensions. To address these issues, the projection correlation is improved by using a modified weight function, which reduces the complexity to O{n2(p+q)}. We estimate the improved projection correlation with U-statistic theory. Importantly, its asymptotic null distribution is standard normal, thanks to the high dimesnionality of the random vectors. This expedites the implementation of independence tests substantially. To enhance the power performance in high dimensions, we propose incorporating a cross-validation procedure with feature screening into the projection correlation test. The implementation efficacy and power enhancement are confirmed through extensive numerical studies.

Keywords: Independence; Nonlinear dependence; Projection correlation (search for similar items in EconPapers)
Date: 2024
References: Add references at CitEc
Citations:

Downloads: (external link)
http://hdl.handle.net/10.1093/biomet/asad070 (application/pdf)
Access to full text is restricted to subscribers.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:oup:biomet:v:111:y:2024:i:3:p:1013-1027.

Ordering information: This journal article can be ordered from
https://academic.oup.com/journals

Access Statistics for this article

Biometrika is currently edited by Paul Fearnhead

More articles in Biometrika from Biometrika Trust Oxford University Press, Great Clarendon Street, Oxford OX2 6DP, UK.
Bibliographic data for series maintained by Oxford University Press ().