Market Efficiency in the Age of Big Data

Ian Martin and Stefan Nagel

No 26586, NBER Working Papers from National Bureau of Economic Research, Inc

Abstract: Modern investors face a high-dimensional prediction problem: thousands of observable variables are potentially relevant for forecasting. We reassess the conventional wisdom on market efficiency in light of this fact. In our model economy, which resembles a typical machine learning setting, N assets have cash flows that are a linear function of J firm characteristics, but with uncertain coefficients. Risk-neutral Bayesian investors impose shrinkage (ridge regression) or sparsity (Lasso) when they estimate the J coefficients of the model and use them to price assets. When J is comparable in size to N, returns appear cross-sectionally predictable using firm characteristics to an econometrician who analyzes data from the economy ex post. A factor zoo emerges even without p-hacking and data-mining. Standard in-sample tests of market efficiency reject the no-predictability null with high probability, despite the fact that investors optimally use the information available to them in real time. In contrast, out-of-sample tests retain their economic meaning.

JEL-codes: C11 G12 G14 (search for similar items in EconPapers)
Date: 2019-12
New Economics Papers: this item is included in nep-big, nep-fmk and nep-ore
Note: AP ME
References: View references in EconPapers View complete reference list from CitEc
Citations: View citations in EconPapers (4)

Published as Ian W.R. Martin & Stefan Nagel, 2021. "Market efficiency in the age of big data," Journal of Financial Economics, .

Downloads: (external link)
http://www.nber.org/papers/w26586.pdf (application/pdf)

Related works:
Journal Article: Market efficiency in the age of big data (2022)
Working Paper: Market efficiency in the age of big data (2022)
Working Paper: Market Efficiency in the Age of Big Data (2019)
Working Paper: Market Efficiency in the Age of Big Data (2019)
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:nbr:nberwo:26586

Ordering information: This working paper can be ordered from
http://www.nber.org/papers/w26586

Access Statistics for this paper

More papers in NBER Working Papers from National Bureau of Economic Research, Inc National Bureau of Economic Research, 1050 Massachusetts Avenue Cambridge, MA 02138, U.S.A.. Contact information at EDIRC.
Bibliographic data for series maintained by ().