Local Processing of Massive Databases with R: A National Analysis of a Brazilian Social Programme

Hellen Paz, Mateus Maia, Fernando Moraes, Ricardo Lustosa, Lilia Costa, Samuel Macêdo, Marcos E. Barreto and Anderson Ara
Additional contact information
Hellen Paz: Statistics Department, Federal University of Bahia, 40.170-110 Salvador-BA, Brazil
Mateus Maia: Statistics Department, Federal University of Bahia, 40.170-110 Salvador-BA, Brazil
Fernando Moraes: Statistics Department, Federal University of Bahia, 40.170-110 Salvador-BA, Brazil
Ricardo Lustosa: Institute of Collective Health, Federal University of Bahia, 40.110-040 Salvador-BA, Brazil
Lilia Costa: Statistics Department, Federal University of Bahia, 40.170-110 Salvador-BA, Brazil
Samuel Macêdo: Department of Natural Sciences and Mathematics, Federal Institute of Pernambuco, 50.740-545 Recife-PE, Brazil
Marcos E. Barreto: Computer Science Department, Federal University of Bahia, 40.170-110 Salvador-BA, Brazil
Anderson Ara: Statistics Department, Federal University of Bahia, 40.170-110 Salvador-BA, Brazil

Stats, 2020, vol. 3, issue 4, 1-21

Abstract: The analysis of massive databases is a key issue for most applications today and the use of parallel computing techniques is one of the suitable approaches for that. Apache Spark is a widely employed tool within this context, aiming at processing large amounts of data in a distributed way. For the Statistics community, R is one of the preferred tools. Despite its growth in the last years, it still has limitations for processing large volumes of data in single local machines. In general, the data analysis community has difficulty to handle a massive amount of data on local machines, often requiring high-performance computing servers. One way to perform statistical analyzes over massive databases is combining both tools (Spark and R) via the sparklyr package, which allows for an R application to use Spark. This paper presents an analysis of Brazilian public data from the Bolsa Família Programme (BFP—conditional cash transfer), comprising a large data set with 1.26 billion observations. Our goal was to understand how this social program acts in different cities, as well as to identify potentially important variables reflecting its utilization rate. Statistical modeling was performed using random forest to predict the utilization rated of BFP. Variable selection was performed through a recent method based on the importance and interpretation of variables in the random forest model. Among the 89 variables initially considered, the final model presented a high predictive performance capacity with 17 selected variables, as well as indicated high importance of some variables for the observed utilization rate in income, education, job informality, and inactive youth, namely: family income, education, occupation and density of people in the homes. In this work, using a local machine, we highlighted the potential of aggregating Spark and R for analysis of a large database of 111.6 GB. This can serve as proof of concept or reference for other similar works within the Statistics community, as well as our case study can provide important evidence for further analysis of this important social support programme.

Keywords: big data; massive databases; impact evaluation; sparklyr; Bolsa Família (search for similar items in EconPapers)
JEL-codes: C1 C10 C11 C14 C15 C16 (search for similar items in EconPapers)
Date: 2020
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
https://www.mdpi.com/2571-905X/3/4/28/pdf (application/pdf)
https://www.mdpi.com/2571-905X/3/4/28/ (text/html)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:gam:jstats:v:3:y:2020:i:4:p:28-464:d:430923

Access Statistics for this article

Stats is currently edited by Mrs. Minnie Li

More articles in Stats from MDPI
Bibliographic data for series maintained by MDPI Indexing Manager ().