Machine Learning for Demand Estimation in Long Tail Markets

Hammaad Adam (), Pu He () and Fanyin Zheng ()
Additional contact information
Hammaad Adam: Institute for Data, Systems, and Society, Massachusetts Institute of Technology, Cambridge, Massachusetts 02142
Pu He: Two Sigma, New York, New York 10013
Fanyin Zheng: Decision, Risk, and Operations, Columbia Business School, New York, New York 10025

Management Science, 2024, vol. 70, issue 8, 5040-5065

Abstract: Random coefficient multinomial logit models are widely used to estimate customer preferences from sales data. However, these estimation models can only allow for products with positive sales; this selection leads to highly biased estimates in long tail markets, that is, markets where many products have zero or low sales. Such markets are increasingly common in areas such as online retail and other online marketplaces. In this paper, we propose a two-stage estimator that uses machine learning to correct for this bias. Our method first uses deep learning to predict the market shares of all products, where the neural network’s structure mirrors the random coefficient multinomial logit model’s data generating process. In the second stage, we use the predictions of the first stage to reweight the observed shares in a way that corrects for the induced bias and maintains the causal interpretation of the structural model. We show that the estimated parameters are consistent in the number of markets. Our method performs well on simulated and real long tail data, producing accurate estimates of customer behavior. These improved estimates can subsequently be used to provide prescriptive policy recommendations on important managerial decisions such as pricing, assortment, and so on.

Keywords: utility-preference: estimation; utility-preference: choice functions; economics: econometrics (search for similar items in EconPapers)
Date: 2024
References: Add references at CitEc
Citations:

Downloads: (external link)
http://dx.doi.org/10.1287/mnsc.2023.4893 (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:inm:ormnsc:v:70:y:2024:i:8:p:5040-5065

Access Statistics for this article

More articles in Management Science from INFORMS Contact information at EDIRC.
Bibliographic data for series maintained by Chris Asher ().