Opportunities and Challenges: Lessons from Analyzing Terabytes of Scanner Data

Ng, Serena

Opportunities and Challenges: Lessons from Analyzing Terabytes of Scanner Data

No 23673, NBER Working Papers from National Bureau of Economic Research, Inc

Abstract: This paper seeks to better understand what makes big data analysis different, what we can and cannot do with existing econometric tools, and what issues need to be dealt with in order to work with the data efficiently. As a case study, I set out to extract any business cycle information that might exist in four terabytes of weekly scanner data. The main challenge is to handle the volume, variety, and characteristics of the data within the constraints of our computing environment. Scalable and efficient algorithms are available to ease the computation burden, but they often have unknown statistical properties and are not designed for the purpose of efficient estimation or optimal inference. As well, economic data have unique characteristics that generic algorithms may not accommodate. There is a need for computationally efficient econometric methods as big data is likely here to stay.

JEL-codes: C55 C81 (search for similar items in EconPapers)
Date: 2017-08
New Economics Papers: this item is included in nep-big and nep-ecm
Note: TWP
References: View references in EconPapers View complete reference list from CitEc
Citations: View citations in EconPapers (20)

Downloads: (external link)
http://www.nber.org/papers/w23673.pdf (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:nbr:nberwo:23673

Ordering information: This working paper can be ordered from
http://www.nber.org/papers/w23673

Access Statistics for this paper

More papers in NBER Working Papers from National Bureau of Economic Research, Inc National Bureau of Economic Research, 1050 Massachusetts Avenue Cambridge, MA 02138, U.S.A.. Contact information at EDIRC.
Bibliographic data for series maintained by ().