Robust and efficient subsampling algorithms for massive data logistic regression

Jun Jin, Shuangzhe Liu and Tiefeng Ma

Journal of Applied Statistics, 2024, vol. 51, issue 8, 1427-1445

Abstract: Datasets that are big with regard to their volume, variety and velocity are becoming increasingly common. However, limitations in computer processing can often restrict analysis performed on them. Nonuniform subsampling methods are effective in reducing computational loads for massive data. However, the variance of the estimator of nonuniform subsampling methods becomes large when the subsampling probabilities are highly heterogenous. To this end, we develop two new algorithms to improve the estimation method for massive data logistic regression based on a chosen hard threshold value and combining subsamples, respectively. The basic idea of the hard threshold method is to carefully select a threshold value and then replace subsampling probabilities lower than the threshold value with the chosen value itself. The main idea behind the combining subsamples method is to better exploit information in the data without hitting the computation bottleneck by generating many subsamples and then combining estimates constructed from the subsamples. The combining subsamples method obtains the standard error of the parameter estimator without estimating the sandwich matrix, which provides convenience for statistical inference in massive data, and can significantly improve the estimation efficiency. Asymptotic properties of the resultant estimators are established. Simulations and analysis of real data are conducted to assess and showcase the practical performance of the proposed methods.

Date: 2024
References: Add references at CitEc
Citations:

Downloads: (external link)
http://hdl.handle.net/10.1080/02664763.2023.2205611 (text/html)
Access to full text is restricted to subscribers.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:taf:japsta:v:51:y:2024:i:8:p:1427-1445

Ordering information: This journal article can be ordered from
http://www.tandfonline.com/pricing/journal/CJAS20

DOI: 10.1080/02664763.2023.2205611

Access Statistics for this article

Journal of Applied Statistics is currently edited by Robert Aykroyd

More articles in Journal of Applied Statistics from Taylor & Francis Journals
Bibliographic data for series maintained by Chris Longhurst ().