How Well Do Structural Demand Models Work? Counterfactual Predictions in School Choice

Parag Pathak and Peng Shi

No 24017, NBER Working Papers from National Bureau of Economic Research, Inc

Abstract: Discrete choice demand models are widely used for counterfactual policy simulations, yet their out-of-sample performance is rarely assessed. This paper uses a large-scale policy change in Boston to investigate the performance of discrete choice models of school demand. In 2013, Boston Public Schools considered several new choice plans that differ in where applicants can apply. At the request of the mayor and district, we forecast the alternatives' effects by estimating discrete choice models. This work led to the adoption of a plan which significantly altered choice sets for thousands of applicants. Pathak and Shi (2014) update forecasts prior to the policy change and describe prediction targets involving access, travel, and unassigned students. Here, we assess how well these ex ante counterfactual predictions compare to actual outcome under the new choice sets. We find that a simple ad hoc model performs as well as the more complicated structural choice models for one of the two grades we examine. However, the structural models' inconsistent performance is largely due to prediction errors in applicant characteristics, which are auxiliary inputs. Once we condition on the actual applicant characteristics, the structural choice models outperform the ad hoc alternative in predicting both choice patterns and policy relevant outcomes. Moreover, refitting the models using the new choice data does not significantly improve their prediction accuracy, suggesting that the choice models are indeed “structural.” Our findings show that structural demand models can effectively predict counterfactual outcomes, as long there are accurate forecasts about auxiliary input variables.

JEL-codes: C10 C78 D12 I20 (search for similar items in EconPapers)
Date: 2017-11
New Economics Papers: this item is included in nep-dcm, nep-des, nep-for and nep-ure
Note: ED IO LS
References: View references in EconPapers View complete reference list from CitEc
Citations: View citations in EconPapers (12)

Published as Parag A. Pathak & Peng Shi, 2020. "How well do structural demand models work? Counterfactual predictions in school choice," Journal of Econometrics, .

Downloads: (external link)
http://www.nber.org/papers/w24017.pdf (application/pdf)

Related works:
Journal Article: How well do structural demand models work? Counterfactual predictions in school choice (2021)
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:nbr:nberwo:24017

Ordering information: This working paper can be ordered from
http://www.nber.org/papers/w24017

Access Statistics for this paper

More papers in NBER Working Papers from National Bureau of Economic Research, Inc National Bureau of Economic Research, 1050 Massachusetts Avenue Cambridge, MA 02138, U.S.A.. Contact information at EDIRC.
Bibliographic data for series maintained by ().