An econometric perspective on algorithmic subsampling

Sokbae (Simon) Lee and Serena Ng ()

No CWP18/20, CeMMAP working papers from Centre for Microdata Methods and Practice, Institute for Fiscal Studies

Abstract: Datasets that are terabytes in size are increasingly common, but computer bottlenecks often frustrate a complete analysis of the data. While more data are better than less, diminishing returns suggest that we may not need terabytes of data to estimate a parameter or test a hypothesis. But which rows of data should we analyze, and might an arbitrary subset of rows preserve the features of the original data? This paper reviews a line of work that is grounded in theoretical computer science and numerical linear algebra, and which ?nds that an algorithmically desirable sketch, which is a randomly chosen subset of the data, must preserve the eigenstructure of the data, a property known as a subspace embedding. Building on this work, we study how prediction and inference can be a?ected by data sketching within a linear regression setup. We show that the sketching error is small compared to the sample size e?ect which a researcher can control. As a sketch size that is algorithmically optimal may not be suitable for prediction and inference, we use statistical arguments to provide ‘inference conscious’ guides to the sketch size. When appropriately implemented, an estimator that pools over di?erent sketches can be nearly as e?cient as the infeasible one using the full sample.

Date: 2020-05-04
References: View references in EconPapers View complete reference list from CitEc
Citations: View citations in EconPapers (6)

Downloads: (external link)
https://www.ifs.org.uk/uploads/CWP1820-An-economet ... hmic-subsampling.pdf (application/pdf)
Our link check indicates that this URL is bad, the error code is: 404 Not Found (https://www.ifs.org.uk/uploads/CWP1820-An-econometric-perspective-on-algorithmic-subsampling.pdf [302 Found]--> https://ifs.org.uk/uploads/CWP1820-An-econometric-perspective-on-algorithmic-subsampling.pdf)

Related works:
Journal Article: An Econometric Perspective on Algorithmic Subsampling (2020)
Working Paper: An Econometric Perspective on Algorithmic Subsampling (2020)
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:ifs:cemmap:18/20

Ordering information: This working paper can be ordered from
The Institute for Fiscal Studies 7 Ridgmount Street LONDON WC1E 7AE

Access Statistics for this paper

More papers in CeMMAP working papers from Centre for Microdata Methods and Practice, Institute for Fiscal Studies The Institute for Fiscal Studies 7 Ridgmount Street LONDON WC1E 7AE. Contact information at EDIRC.
Bibliographic data for series maintained by Emma Hyman ().