Scaling by subsampling for big data, with applications to statistical learning

Patrice Bertail, Mohammed Bouchouia, Ons Jelassi, Jessica Tressou and Mélanie Zetlaoui

Journal of Nonparametric Statistics, 2024, vol. 36, issue 1, 78-117

Abstract: Handling large datasets and calculating complex statistics on huge datasets require important computing resources. Using subsampling methods to calculate statistics of interest on small samples is often used in practice to reduce computational complexity, for instance using the divide and conquer strategy. In this article, we recall some results on subsampling distributions and derive a precise rate of convergence for these quantities and the corresponding quantiles. We also develop some standardisation techniques based on subsampling unstandardised statistics in the framework of large datasets. It is argued that using several subsampling distributions with different subsampling sizes brings a lot of information on the behaviour of statistical learning procedures: subsampling allows to estimate the rate of convergence of different algorithms, to estimate the variability of complex statistics, to estimate confidence intervals for out-of-sample errors and interpolate their values at larger scales. These results are illustrated on simulations, but also on two important datasets, frequently analysed in the statistical learning community, EMNIST (recognition of digits) and VeReMi (analysis of Network Vehicular Reference Misbehavior).

Date: 2024
References: Add references at CitEc
Citations:

Downloads: (external link)
http://hdl.handle.net/10.1080/10485252.2023.2219782 (text/html)
Access to full text is restricted to subscribers.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:taf:gnstxx:v:36:y:2024:i:1:p:78-117

Ordering information: This journal article can be ordered from
http://www.tandfonline.com/pricing/journal/GNST20

DOI: 10.1080/10485252.2023.2219782

Access Statistics for this article

Journal of Nonparametric Statistics is currently edited by Jun Shao

More articles in Journal of Nonparametric Statistics from Taylor & Francis Journals
Bibliographic data for series maintained by Chris Longhurst ().