Econometrics at scale: Spark up big data in economics

Benjamin Bluhm and Jannic Cutura

No 266, SAFE Working Paper Series from Leibniz Institute for Financial Research SAFE

Abstract: This paper provides an overview of how to use \big data" for economic research. We investigate the performance and ease of use of different Spark applications running on a distributed file system to enable the handling and analysis of data sets which were previously not usable due to their size. More specifically, we explain how to use Spark to (i) explore big data sets which exceed retail grade computers memory size and (ii) run typical econometric tasks including microeconometric, panel data and time series regression models which are prohibitively expensive to evaluate on stand-alone machines. By bridging the gap between the abstract concept of Spark and ready-to-use examples which can easily be altered to suite the researchers need, we provide economists and social scientists more generally with the theory and practice to handle the ever growing datasets available. The ease of reproducing the examples in this paper makes this guide a useful reference for researchers with a limited background in data handling and distributed computing.

Keywords: Econometrics; Distributed Computing; Apache Spark (search for similar items in EconPapers)
JEL-codes: C53 C55 (search for similar items in EconPapers)
Date: 2020
New Economics Papers: this item is included in nep-big and nep-ecm
References: View references in EconPapers View complete reference list from CitEc
Citations: View citations in EconPapers (3)

Downloads: (external link)
https://www.econstor.eu/bitstream/10419/214153/1/1690458135.pdf (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:zbw:safewp:266

DOI: 10.2139/ssrn.3226976

Access Statistics for this paper

More papers in SAFE Working Paper Series from Leibniz Institute for Financial Research SAFE Contact information at EDIRC.
Bibliographic data for series maintained by ZBW - Leibniz Information Centre for Economics (econstor@zbw-workspace.eu).