Large-Scale Global and Simultaneous Inference: Estimation and Testing in Very High Dimensions

T. Tony Cai () and Wenguang Sun ()
Additional contact information
T. Tony Cai: Department of Statistics, The Wharton School, University of Pennsylvania, Philadelphia, Pennsylvania 19104
Wenguang Sun: Department of Data Sciences and Operations, Marshall School of Business, University of Southern California, Los Angeles, California 90089

Annual Review of Economics, 2017, vol. 9, issue 1, 411-439

Abstract: Due to rapid technological advances, researchers are now able to collect and analyze ever larger data sets. Statistical inference for big data often requires solving thousands or even millions of parallel inference problems simultaneously. This poses significant challenges and calls for new principles, theories, and methodologies. This review provides a selective survey of some recently developed methods and results for large-scale statistical inference, including detection, estimation, and multiple testing. We begin with the global testing problem, where the goal is to detect the existence of sparse signals in a data set, and then move to the problem of estimating the proportion of nonnull effects. Finally, we focus on multiple testing with false discovery rate (FDR) control. The FDR provides a powerful and practical approach to large-scale multiple testing and has been successfully used in a wide range of applications. We discuss several effective data-driven procedures and also present efficient strategies to handle various grouping, hierarchical, and dependency structures in the data.

Keywords: compound decision problem; dependence; detection boundary; false discovery rate; global inference; multiple testing; null distribution; signal detection; simultaneous inference; sparsity (search for similar items in EconPapers)
JEL-codes: C12 C13 C44 C55 G11 (search for similar items in EconPapers)
Date: 2017
References: Add references at CitEc
Citations: View citations in EconPapers (3)

Downloads: (external link)
https://doi.org/10.1146/annurev-economics-063016-104355 (application/pdf)
Full text downloads are only available to subscribers. Visit the abstract page for more information.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:anr:reveco:v:9:y:2017:p:411-439

Ordering information: This journal article can be ordered from
http://www.annualreviews.org/action/ecommerce

Access Statistics for this article

More articles in Annual Review of Economics from Annual Reviews Annual Reviews 4139 El Camino Way Palo Alto, CA 94306, USA.
Bibliographic data for series maintained by http://www.annualreviews.org ().